Source-linked AI summary
AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations
Xiangyu Zhao, Chong Wang, Ming Chen, Xudong Zheng, Xiaobing Liu, Jiliang Tang
TL;DR
DLRSs commonly assign every user and item one fixed embedding size, although popularity varies and changes in streaming settings, making manual dimension selection difficult. AutoEmb uses controller networks and soft selection to choose dimensions dynamically; benchmark results report improved recommendation performance and training efficiency, especially with insufficient interaction data.
Problem
Most DLRSs use one fixed embedding size, although user/item popularity varies and changes, making manual dimension assignment difficult at scale in streaming recommendations.
Source: §1 · p. 2Method
AutoEmb uses separate controller networks informed by current popularity and contextual signals to select user/item embedding sizes, with soft selection in an end-to-end differentiable AutoML framework.
Source: §2.3 · p. 4, §2.4 · p. 4Results
On widely used benchmark datasets, AutoEmb significantly improves recommendation performance with highly dynamic popularity and dramatically enhances training efficiency, especially when interaction data is insufficient.
Source: §5 · p. 9Takeaways & Limitations
The reported training-efficiency enhancement is especially pronounced when interaction data is insufficient.
Source: §5 · p. 9Soft selection requires a larger embedding space; developing an end-to-end framework with hard selection remains future work.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Deep learning based recommender systems (DLRSs) often have embedding layers, which are utilized to lessen the dimensionality of categorical variables (e.g. user/item identifiers) and meaningfully transform them in the low-dimensional space. The majority of existing DLRSs empirically pre-define a fixed and unified dimension for all user/item embeddings. It is evident from recent researches that different embedding sizes are highly desired for different users/items according to their popularity. However, manually selecting embedding sizes in recommender systems can be very challenging due to the large number of users/items and the dynamic nature of their popularity. Thus, in this paper, we propose an AutoML based end-to-end framework (AutoEmb), which can enable various embedding dimensions according to the popularity in an automated and dynamic manner. To be specific, we first enhance a typical DLRS to allow various embedding dimensions; then we propose an end-to-end differentiable framework that can automatically select different embedding dimensions according to user/item popularity; finally we propose an AutoML based optimization algorithm in a streaming recommendation setting. The experimental results based on widely used benchmark datasets demonstrate the effectiveness of the AutoEmb framework.
1 INTRODUCTION
DLRSs commonly use one fixed embedding size despite user/item popularity varying; preliminary results indicate size preferences change with interaction count. AutoEmb targets automated, dynamic size selection under streaming conditions.
1 INTRODUCTION
- As interaction count rises, all tested dimensions improve, while smaller embeddings lead initially and larger ones later outperform them in MSE and accuracy.The preliminary study varies training interactions and compares embedding dimensions 2, 16, and 128.Source: Figure 1 · p. 1, §1 · p. 2
- DLRSs map categorical identifiers into dense embeddings, then hidden layers transform them before output layers predict recommendation tasks.Embeddings reduce categorical-variable dimensionality and represent users and items in a latent space.Source: §1 · p. 1
- Smaller embeddings use fewer parameters and can work well with little popularity, whereas larger embeddings need sufficient data but can encode more information as popularity increases.The paper argues that these differing data and capacity needs support allowing different dimensions for users and items.Source: §1 · p. 2
- AutoEmb addresses the difficulty of manually assigning dimensions at scale under dynamic popularity and the fixed input dimensions expected by existing hidden layers.It is proposed as an end-to-end differentiable AutoML framework for automated, dynamic dimension selection.Source: §1 · p. 2
2 FRAMEWORK
AutoEmb extends a typical recommender to support popularity-dependent embedding dimensions, then uses differentiable selection and AutoML-based optimization to adapt them in streaming recommendations.
2.1 A Basic DLRS Architecture
- A basic DLRS maps user and item IDs to embeddings, transforms them through fully connected hidden layers, and predicts recommendation outcomes.Its fixed architecture cannot handle varying embedding dimensions.Source: §2.1 · p. 2
2.2 The Enhanced DLRS Model
- Different embedding spaces are transformed to a common dimension so the model can select representations according to user and item popularity.BatchNorm and Tanh make the transformed embeddings magnitude-comparable.Source: §2.2 · p. 3
2.3 The Controller
- Separate user and item controllers use current popularity and contextual information, including previous hyperparameters and their losses, to assign probabilities to embedding spaces.The controller selects the space with the largest probability.Source: §2.3 · p. 4
2.4 Soft Selection
- Soft selection forms each representation as a probability-weighted sum of transformed embeddings, making the enhanced DLRS end-to-end differentiable.The user and item representations are controlled separately.Source: §2.4 · p. 4
2.5 An Optimization Method
- A DARTS-based bilevel procedure updates DLRS parameters using training loss and controller parameters using validation loss.The approximation estimates the DLRS parameters with one training step; the method omits discrete architecture selection because popularity changes with new interactions.Source: §2.5 · pp. 4–5
- In batch-based streaming, AutoEmb predicts and evaluates each full mini-batch using current parameters before minimizing prediction-label loss; prior user-item interactions are sampled for validation, and parameters update during offline estimation and online inference.Source: §2.5 · p. 5
3 EXPERIMENTS
Experiments compare AutoEmb with representative baselines across three datasets and examine how popularity and training-data availability relate to performance. They also describe the experimental setup and evaluate regression and classification tasks.
3.1 Datasets
- Evaluation uses Movielens-20m, Movielens-latest, and Netflix Prize data, with 70% of interactions for offline parameter estimation and 30% for online learning.Other contextual features, such as users’ age and items’ category, are excluded to isolate embedding selection.Source: §3.1 · pp. 5–6
3.4 Overall Performance in the Online Stage
- AutoEmb outperforms representative baselines across different datasets and metrics in the online stage.The comparison includes Table 2; the supplied passages do not provide its cell values.Source: Table 2 · p. 6, §3.4 · pp. 6–7
3.2 Implement Details
- DLRS uses three embedding dimensions [2, 16, 128], concatenates each user/item’s embeddings, and has a controller with a 512 × 3 Softmax output generating weights for the three embedding sizes; batch size is 500.Source: §3.2 · p. 6
3.3 Evaluation Metrics
- The experiments include binarized-rating regression evaluated by MSE loss and accuracy, and five-class rating classification evaluated by crossentropy and accuracy.Regression uses 0.5 as the threshold for assigning labels.Source: §3.3 · p. 6
3.5 Performance with Popularity
- AutoEmb outperforms FSE and SAM across popularity levels, while its controller’s weights shift from smaller to larger embeddings as popularity increases.At low popularity, FSE performs worse than SAM and AutoEmb; with higher popularity, FSE outperforms SAM.Source: §3.5 · p. 7, Figure 6 · p. 7, Figure 7 · p. 7
3.6 Performance with Data
- AutoML outperforms SAM and FSE throughout training, with especially strong gains early when training examples are insufficient.SAM performs worst early because its controller overfits to insufficient examples; that overfitting gradually eases as more data arrives.Source: §3.6 · pp. 7–8, Figure 8 · p. 8
4 RELATED WORK
Related work covers deep learning recommender systems and neural architecture search. The paper positions its focus on embedding layers and notes that existing multi-size embedding work does not fit streaming settings with dynamic popularity.
Deep learning based recommender systems
- Deep learning recommender work includes models such as NCF [9] and DeepFM [8], while most prior work emphasizes network architectures rather than embedding layers.The reviewed approaches also include recommendation methods using side information, autoencoders, CNNs, and recurrent networks.Source: §4 · p. 8
AutoML for Neural Architecture Search
- Neural architecture search uses approaches including controller-based sampling, hypernetworks, and differentiable optimization; Multi-size Embedding [12] cannot address streaming settings with unknown, dynamic popularity.The passage also describes search-space reduction and transfer learning as NAS directions.Source: §4 · p. 8
5 CONCLUSION
AutoEmb automatically selects user and item embedding dimensions according to popularity in streaming recommendations. On benchmark datasets, it improves recommendation performance under highly dynamic popularity and boosts training efficiency, especially with insufficient interaction data.
5 CONCLUSION
- AutoEmb augments a DLRS to accept varying embedding dimensions and uses an AutoML controller to select dimensions based on current user or item popularity.The framework is intended to automate and dynamically adapt dimension selection in streaming recommendation.Source: §5 · p. 8
- AutoEmb significantly improves recommendation performance with highly dynamic popularity and enhances training efficiency, especially when interaction data is insufficient.The results are based on widely used benchmark datasets.Source: §5 · p. 9
- Soft dimension selection makes the framework end-to-end differentiable but requires a larger embedding space; hard selection is proposed as future work.Source: §5 · p. 9
- Future directions include designing the whole DLRS architecture, applying the framework to more categorical features, and incorporating non-categorical features.Source: §5 · p. 9