NeuSE: A Neural Snapshot Ensemble Method for Collaborative Filtering

• Global empirical-risk optimization can leave some users or items overfit or underfit, yielding suboptimal CF performance.Source: §1 · pp. 1–2
• Adaptive learning rates cannot explore other local optima, Bayesian methods face prior-selection and computational challenges, and ensembles can be costly [19, 20].NeuSE uses snapshots from the global learning trajectory to enable ensemble learning with little additional training cost.Source: §1 · p. 2
• Up to 15.9% relatively, NeuSE improves existing collaborative filtering methods across almost all algorithm and recommendation-task types.Source: §1 · p. 2
• NeuSE combines training snapshots using model-, user-, and item-memory networks to produce adaptive ensemble weights for each user-item pair.A fully connected layer combines the three memory-network outputs.Source: §1 · p. 2
• Its soft-label learning method lowers loss when a predicted snapshot is close to the optimal snapshot and raises it otherwise.Source: §1 · p. 2

2 RELATED WORK

• Because CF is non-convex, globally optimized models may not be optimal for each data subset, while ensembles can be costly on large datasets [12] or complex neural methods [9].Existing ensemble methods may also be specific to particular CF algorithms; LLORMA is described as applicable to matrix factorization but not UserKNN or ItemKNN.Source: §2 · p. 2
• The work aims to improve almost all kinds of existing CF algorithms without significantly increasing training overhead.Source: §2 · p. 3
• Adaptive learning rates do not explore different local optima, Dropout is limited to neural CF, and Bayesian approaches face prior-distribution and scalability issues.The proposed method is described as orthogonal to these techniques and combinable with them.Source: §2 · p. 3
• Prior trajectory ensembles average snapshot outputs [41] or use cyclic learning rates to reach different minima [12], but averaging can include poor-generalizing models.Source: §2 · p. 3
• Xie et al. [41] learn global snapshot weights, whereas NeuSE adapts weights for each user-item pair to capture users' differing preferences across items.The paper reports that a simple MLP achieves performance similar to simple averaging of snapshot outputs.Source: §2 · p. 3

3.1 Preliminaries

• CF methods learn user and item parameters by minimizing a loss over the dataset, then use an interaction function to generate recommendation scores.The examples are dot product for matrix factorization and a multilayer perceptron for NeuMF [9].Source: §3.1 · p. 3
• The loss is often non-convex in user and item parameters jointly, allowing many local optima; guarantees for matrix factorization under symmetric positive definite assumptions do not cover general asymmetric CF rating matrices.The passage names SGD, ALS, and MCMC as iterative learning methods.Source: §3.1 · p. 3

3.2.2 Model Selection.

• Around 0.3% relative improvement came from averaging the properly selected set 2 snapshots, whereas noisy set 1 performed worse than the optimal single model.The figure compares the single optimal snapshot, snapshots every 50 epochs through 500, and snapshots at epochs 100, 200, and 300.Source: Figure 2 · p. 4, §3.2.2 · p. 5

3.2 Case Studies

• The motivating case studies use MovieLens 1M, a random 9:1 train-test split, and RSVD for rating prediction in Figures 1 and 2.Source: §3.2 · p. 4, Figure 1 · p. 4, Figure 2 · p. 4

3.2.1 Global vs. Local Convergence.

• Around epoch 175, the global optimum favored users with 50–500 ratings, while users with fewer than 50 or more than 500 ratings faced underfitting or overfitting; about 90% of ratings involved varying optimal snapshots across user-item pairs.The trends were also observed with AdaGrad SGD, and the study concludes that different user-item pairs may need different snapshot models.Source: §3.2.1 · p. 4, Figure 1 · p. 4
• AdaGrad SGD made test errors more stable across user groups, but its optimal global model had almost the same RMSE as vanilla SGD and did not fundamentally solve suboptimal performance.The paper says adaptive learning rates can make it easier to identify a near-optimal epoch range, without guaranteeing better performance.Source: §3.2.1 · p. 4, Figure 1 · p. 4

3.2.2 Model Selection.

• Snapshot selection is crucial because poor base models can degrade ensembles, and selecting snapshots from a stable epoch range [41] is unsuitable when no such range exists.The passage identifies snapshot model selection as crucial to existing methods.Source: §3.2.2 · p. 4, Figure 1 · p. 4

4 NEUSE: NEURAL SNAPSHOT ENSEMBLE

• NeuSE selects snapshots at fixed epoch intervals (or from intermediate non-learning-based CF models) and adaptively weights their predictions for each user-item pair using global, user, and item memories with soft labels.Soft labels give nearby snapshots of an optimal model more weight than snapshots farther away.Source: §4 · p. 5

4.1 Network Architecture

• The memory networks address overfitting from sparse individual user or item ratings by combining global information with memories of each user's and item's top-k neighbors.The architecture has input, memory, and output layers.Source: §4.1 · p. 5

4.1.1 Input Layer.

• One-hot user and item identities are projected into dense d-dimensional embeddings, concatenated into a 2d-dimensional user-item vector, and passed to the memory layer.The user and item embeddings are produced by fully connected layers.Source: §4.1.1 · p. 5

4.1.2 Memory Layer.

• Model memory uses internal slots to assess snapshot relevance to a user-item pair and external slots to represent snapshots; attention weights combine snapshot representations, while user and item memories aggregate neighbor information.The user memory considers users who interacted with the target item, and the item memory considers items interacted with by the target user.Source: §4.1.2 · pp. 6–7

4.1.3 Output Layer.

• The output layer combines snapshot predictions, model/user/item memory representations, and corresponding bias terms, then applies a softmax layer to produce ensemble weights.The biases are intended to capture distinct characteristics of the data, user, and item.Source: §4.1.3 · pp. 7–8

4.1.4 Multi-Layer Memory Networks.

• Multi-layer memory networks pass each hop's output to the next, allowing snapshot representations and user/item neighborhood information to be refined using one another across layers.The final hop produces ensemble weights as in the single-hop network.Source: §4.1.4 · pp. 8–9, Figure 4 · p. 8

4.2 Loss Function with Soft Labels

• NeuSE frames ensemble learning as multi-class classification with soft targets that favor snapshots near the optimum, trains with KL divergence, and averages snapshot scores using the predicted weights.The soft targets reduce the risk of selecting one uncertain optimum when individual users or items have limited ratings; training uses SGD.Source: §4.2 · p. 9

5.1 Efficiency Analysis

• NeuSE adds O(DHN_R) computation per epoch and memory overhead versus NeuMF, but faster convergence makes its overall training time much lower.D is the memory-network embedding size, H the number of hops, and N_R the number of ratings.Source: §5.1 · pp. 9–10

5.2 Addressing Cold-start User/item

• Unseen users with ratings on seen items and unseen items rated by seen users can be handled by replacing one-hot inputs with rating vectors.For users who rate only unseen items, retraining or auxiliary information such as demographics or genres may be needed.Source: §5.2 · p. 10

5.3 Application Scope

• For non-learning methods, NeuSE fails when snapshots cannot be created, as with most-popular recommendation, but can improve methods such as KNN when snapshots are available.The discussion gives different neighbor counts as one way to create KNN snapshots.Source: §5.3 · p. 10
• For learning-based CF, non-convex iterative optimization provides intermediate snapshots, whereas closed-form convex methods do not; convex CF is rare, so NeuSE can apply to most learning-based methods.The passage identifies convex optimization methods [34] as cases without snapshots.Source: §5.3 · p. 10
• NeuSE can improve collaborative-filtering performance when snapshot models can be created, as is possible for most popular algorithms.The discussion frames snapshot availability as the condition for applying NeuSE.Source: §5.3 · p. 10

6.1.1 Dataset Description.

• Three real-world datasets support both explicit ratings and implicit positive-feedback evaluation, with rating scales of 1–5 for MovieLens and Amazon and 1–10 for Book Crossing.Dataset details are listed in Table 1.Source: §6.1.1 · p. 11

6.1.2 Evaluation Protocol.

• Evaluation uses chronological leave-one-out splits [9], selecting each user’s latest rating for testing and penultimate rating for validation, with fixed sets of 99 randomly selected negatives per test case for top-N recommendation [9].Models are selected by validation performance before testing.Source: §6.1.2 · p. 11

6.1.3 Compared Methods.

• NeuSE is compared with the best-validation single snapshot, horizontal averaging [41], horizontal stacked ensembles [41] using an MLP, and cyclic-learning-rate snapshot ensembles [12].The cyclic-learning-rate method applies only to models learned via SGD.Source: §6.1.3 · p. 11

6.1.4 Baseline CF Methods.

• The study compares snapshot ensembles using six collaborative-filtering algorithms spanning memory-based, factorization, and neural methods across rating prediction and top-N recommendation.The algorithms are UserKNN, ItemKNN, RSVD, FM, AutoRec, and NeuMF.Source: §6.1.4 · p. 11

6.1.5 Hyperparameters.

• NeuSE uses batch size 128, learning rate 0.01, Adam [14], dropout 0.5, ReLU, 16-dimensional user/item embeddings, Gaussian initialization N(0, 0.01), and a two-hop memory network.Baseline hyperparameters remain consistent across snapshot ensemble methods; snapshot selection uses epoch variations of 10 for RSVD, FM, and AutoRec and 5 for NeuMF through 200 epochs, or neighbor counts from 10 to 100 for UserKNN and ItemKNN. Baseline embedding or hidden-factor sizes are 8, 8, 500, and 32 for RSVD, FM, AutoRec, and NeuMF, respectively.Source: §6.1.5 · pp. 11–12

6.1.6 Evaluation Metrics.

• For rating prediction, the evaluation metric is RMSE, where lower values indicate better performance.It is computed over user-item pairs in the test set.Source: §6.1.6 · p. 12
• For top-N recommendation, HR@N and NDCG@N measure performance, with higher values indicating better results.NDCG uses discounted gain based on relevance in the ranked top-N list and ideal discounted gain for perfect ranking.Source: §6.1.6 · p. 12

6.2 Recommendation Accuracy

• NeuSE adaptively weights snapshot models for individual ratings, using memory networks to combine their global importance with user- and item-specific characteristics and neighborhood information.This can alleviate the effects of poor models and use information from snapshots that converge to different local optima.Source: §6.2 · pp. 12–13
• Simple averaging improves on the optimal snapshot in only three of six baseline algorithms, while HSE usually beats averaging but cannot memorize user- and item-specific characteristics.SE is comparable to averaging on factorization methods and slightly better on neural-network models; cyclic learning rates may lead neural models to different local optima [12].Source: §6.2 · p. 12
• ∼15.9% over HSE at NDCG for UserKNN on Amazon is the largest reported gain; NeuSE improves over all three snapshot-ensemble methods across baseline algorithms and datasets.Other reported gains are ∼7.5% over the single model at NDCG for UserKNN on Amazon, ∼15.4% over Average at NDCG for ItemKNN on Book Crossing, and ∼4.1% over SE at NDCG for NeuMF on Book Crossing.Source: §6.2 · p. 12, Table 2 · p. 13
• NeuSE improves accuracy for models trained with iterative optimization, Bayesian approaches, and memory-based methods, supporting its use with almost all existing collaborative-filtering algorithms.The reported examples include SGD, MCMC, UserKNN, and ItemKNN.Source: §6.2 · pp. 13–14

6.3.1 Memory Network Layers.

• Two-layer memory networks achieve the highest accuracy across RSVD, FM, and NeuMF, outperforming one- and three-layer networks.One layer cannot use global or neighborhood information from previous layers; three layers are prone to overfitting from additional parameters.Source: §6.3.1 · p. 14

6.3.2 Soft vs. Hard Label.

• ∼0.5% in RSVD, ∼0.2% in FM, and ∼1.4% in NeuMF: soft labels improve accuracy over hard labels.Soft labels adapt losses to snapshot models with different performance, whereas hard labels can overfit noisy individual user/item behavior.Source: §6.3.2 · p. 14

6.3.4 Embedding Size.

• Embedding size 16 yields the highest NeuSE accuracy, while sizes 32 and 64 show signs of overfitting.The results indicate that NeuSE can perform well with lightweight models, supporting fast computation and memory efficiency.Source: Figure 7 · p. 15, §6.3.4 · p. 15

6.3.5 Activation Function.

• ReLU performs slightly better than Sigmoid and Tanh on average, while Tanh performs worst with two- or three-hop memory networks.With one hop, the three activation functions perform similarly; the paper suggests reduced gradient vanishing and better convergence as possible reasons for ReLU’s advantage.Source: Figure 8 · p. 15, §6.3.5 · pp. 15–16

6.3.3 Model Complexity.

• NeuSE consistently improves over the best Single models across different underlying CF model complexities, including both high- and low-capacity settings.The results suggest that NeuSE helps whenever the underlying model has a suboptimal-performance issue.Source: §6.3.3 · p. 15

6.3.7 Efficiency Analysis.

• NeuSE’s training time is comparable to RSVD and FM and lower than NeuMF, so it does not significantly increase CF training time.The comparison uses ReLU and embedding size 16; the paper contrasts NeuSE with ensemble methods that train tens of base models.Source: Figure 10 · p. 16, §6.3.7 · p. 16
• Among snapshot ensemble methods, NeuSE achieves the best performance without significantly increasing overhead relative to HSE and SE; simple averaging requires little overhead.The paper reports that NeuSE’s computation time increases linearly with the number of memory-network hops.Source: Figure 11 · p. 16, §6.3.7 · pp. 16–17

6.3.6 Snapshot Selection.

• Performance varies little across 5–10 snapshot models, with 7–9 performing slightly better; selecting 10 may hurt accuracy, but degradation remains limited.Snapshots are selected from intermediate RSVD models trained every 10 epochs; the authors attribute robustness to NeuSE’s ensemble-selection capability.Source: §6.3.6 · p. 16, Figure 9 · p. 16

6.4 Performance on Large Dataset

• The Netflix Prize dataset contains 100,480,507 ratings from 480,189 users across 17,770 items.NeuSE was evaluated for accuracy and efficiency on top of RSVD, FM, and NeuMF.Source: §6.4 · p. 17
• On Netflix Prize, NeuSE outperforms Average, HSE, and SE across RSVD, FM (SGD), and NeuMF settings.The paper says this result is consistent with its observations on smaller datasets.Source: §6.4 · p. 17
• NeuSE’s training overhead is on par with a baseline CF model on Netflix Prize, while Average is much more efficient because it requires no learning steps.The paper concludes that NeuSE’s training time is comparable to popular CF algorithms on both small and large datasets.Source: §6.4 · p. 17, Figure 12 · p. 18

7 CONCLUSION AND FUTURE WORK

• NeuSE integrates snapshot models from global training through multi-memory networks, adding little computational overhead while improving almost all kinds of existing collaborative-filtering algorithms.The conclusion characterizes these performance improvements as extensive and significant.Source: §7 · p. 18
• The authors propose applying NeuSE beyond collaborative filtering, for example to non-convex neural networks [6].Source: §7 · p. 18