Source-linked AI summary
Optimizing Sequential Recommendation Models with Scaling Laws and Approximate Entropy
Tingjia Shen, Hao Wang, Chuhan Wu, Jin Yao Chin, Wei Guo, Yong Liu, Huifeng Guo, Defu Lian, Ruiming Tang, Enhong Chen
TL;DR
Scaling laws offer a way to study model performance as size increases, but recommendation models pose challenges because performance is more relevant than loss and their data has important quality characteristics. The paper fits HR and NDCG for transformer-based sequential recommendation models and introduces Approximate Entropy as a data-quality measure. It reports predictions across dataset scales and model sizes, with strong correlation in large sequential models.
Problem
Applying existing scaling laws to recommendation models is limited by the relevance of performance rather than loss and by recommendation data's structural, collaborative, noisy, and redundant characteristics.
Source: §1 · p. 2Method
The paper fits HR and NDCG for transformer-based sequential recommendation models and introduces Approximate Entropy to incorporate data quality into its Performance Law.
Source: §1 · p. 2, §3.2.2 · p. 3Results
Takeaways & Limitations
The framework is presented as a way to balance computational resources and pursue optimal performance for a given model configuration.
Source: §1 · p. 2, §6 · p. 9The theorem assumes user sequences can be modeled as a first-order aperiodic stationary Markov chain.
Source: §3.3 · p. 4
Abstract
from arXiv · showhide
Scaling Laws have emerged as a powerful framework for understanding how model performance evolves as they increase in size, providing valuable insights for optimizing computational resources. In the realm of Sequential Recommendation (SR), which is pivotal for predicting users' sequential preferences, these laws offer a lens through which to address the challenges posed by the scalability of SR models. However, the presence of structural and collaborative issues in recommender systems prevents the direct application of the Scaling Law (SL) in these systems. In response, we introduce the Performance Law for SR models, which aims to theoretically investigate and model the relationship between model performance and data quality. Specifically, we first fit the HR and NDCG metrics to transformer-based SR models. Subsequently, we propose Approximate Entropy (ApEn) to assess data quality, presenting a more nuanced approach compared to traditional data quantity metrics. Our method enables accurate predictions across various dataset scales and model sizes, demonstrating a strong correlation in large SR models and offering insights into achieving optimal performance for any given model configuration.
1. Introduction
Applying scaling laws to recommendation models is challenged by the importance of performance metrics and by data quality. The paper proposes a Performance Law for sequential recommendation, using Approximate Entropy to account for data quality and predict model performance.
1. Introduction
- Recommendation scaling laws face a performance-versus-loss mismatch and have not accounted for the structural, collaborative, noisy, and redundant characteristics of recommendation data.The authors identify performance as the more pertinent outcome and data quality as a critical factor influencing outcomes.Source: §1 · p. 2
- The Performance Law fits HR and NDCG for sequential recommendation models and introduces Approximate Entropy to represent data quality alongside data scale.The authors describe replacing data scale with data scale/approximate entropy as a more accurate substitute for training-token count.Source: §1 · p. 2
- The authors report accurate predictions across dataset scales and model sizes, strong correlation in large sequential models, and optimal-performance estimates for a given model size.They also describe using a few initial small-scale training points to infer the impact of model layers and item embedding dimensions.Source: §1 · p. 2
2. Related Work
Sequential recommendation research spans recurrent, attention-based, and other neural approaches, while scaling-law work has studied how model size and training data relate to performance. Prior results also report that performance can decline with model size under overfitting.
2.1. Sequential Recommendation
- Sequential recommendation research spans CNN, GNN, MLP, diffusion, Fourier, RNN, and attention-based methods, including transformer models SASRec and BERT4Rec.The section notes that large models and datasets have improved recommendation performance, motivating study of performance as model size scales.Source: §2.1 · p. 2
2.2. Scaling Law on Large Sequential Models
- Scaling-law studies model loss or performance against model size and training data, but prior work also reports performance decline associated with overfitting.The cited literature covers compute constraints, data limitations, regeneration, and over-training.Source: §2.2 · p. 2
3. Preliminary and Theorem
This section defines sequential recommendation as next-item prediction and adapts scaling-law analysis to account for recommendation-specific structure and data quality. It develops a performance fit whose data parameter combines token count with adjusted approximate entropy.
3.1. Problem Definition
- Sequential recommendation models the probability of the next item from a user’s ordered interaction history.The task is framed around predicting the subsequent item from prior interactions.Source: §3.1 · p. 3
3.2.1. SCALING LAW ON LARGE SEQUENTIAL MODELS
- Directly transferring language-model scaling laws to recommendations is complicated by differences between language and recommendation sequences.The section motivates refining scaling laws to better fit the distribution characteristics of sequential recommendation.Source: §3.2.1 · p. 3
3.2.2. APEN EXTENSION ON RECOMMENDATION
- Approximate Entropy (ApEn) is introduced to adapt scaling laws to recommendation data with structural differences and varying user preferences.The section defines ApEn as a measure of time-series regularity and unpredictability, then uses ApEn′ = 1/ApEn as its final measure because ApEn trends positively with duplication rate.Source: §3.2.2 · p. 3
3.3. Theorem of Performance Law
- The performance fit uses HR@10 and NDCG@10, assumes performance is linear in loss, and adds a log-decay term for large models.The decay term addresses growth in model layers and embedding dimension; the text notes a condition associated with overfitting in the proof discussion.Source: §3.3 · pp. 3, 5
- The data parameter D′ is token count multiplied by ApEn′, while model size is represented by layer count N and item embedding dimension demb.The theorem’s encoding-length analysis assumes user sequences follow a first-order aperiodic stationary Markov chain.Source: §3.3 · p. 4
4. Methodology
The methodology uses decoder-only Transformer models for sequential recommendation, with modified spatial aggregation and pointwise transformation layers. It then retrieves candidate items by similarity to the predicted item representation.
4. Methodology
- The backbone is a decoder-only Transformer that embeds each user’s item and rating sequence before stacking decoder blocks.The study modifies the SiLU activation and Rab positional encoding modules.Source: §4 · p. 5
- Spatial aggregation replaces softmax with SiLU and adds relative attention bias to incorporate positional and temporal information.The passage attributes computational efficiency and stability to removing the need for full normalization.Source: §4 · p. 6
- The pointwise transformation combines gating weights with normalized attention values using a Hadamard product, then applies a single-layer MLP.The gate selectively emphasizes or weakens representation entries.Source: §4 · p. 6
- The predicted item representation is compared with items in the pool to retrieve candidates, using inner-product similarity with sampled negatives.The passage introduces a retrieval-task loss after describing this candidate selection step.Source: §4 · p. 6
5. Experimental Evaluation
Experiments across four datasets test scaling-law fits, performance-law predictions, and model-optimization applications. Results report strong fit correlations, consistent performance-law findings across diverse data, and fitted performance maxima under global and constrained searches.
5.1. Datasets
- The dataset evaluation used three public datasets—MovieLens-1M, Amazon Books, and KuaiRand-pure—and the proprietary Huawei Music dataset.MovieLens-1M contains 1 million ratings from 6,000 users on 4,000 movies.Source: §5.1 · p. 6
5.2. Recommendation Fit on Scaling Law
- All R2 values exceeded 0.95 for KuaiRand-1K fits at maximum sequence lengths of 100, 50, and 25; fitted D′ grew linearly with Tokens/ApEn across all four datasets.The authors report that these results support the scaling-law trend and their predicted relationship between data parameters and dataset size adjusted by ApEn.Source: §5.2 · pp. 6–7, Figure 2 · p. 7, Figure 4 · p. 8
5.3. Recommendation Fit on Performance Law
- Performance-law analyses found consistent quality improvements across diverse datasets and robust correlations between ApEn-based data-quality measures and output performance.The reported relationship held across dataset configurations, including varied sequence lengths and alternative data structures.Source: §5.3 · p. 7
5.4.1. APPLICATION 1: GLOBAL AND LOCAL OPTIMAL PARAMETER SEARCH
- The Performance Law formulation produced performance maxima both globally and under model-size constraints, using demb = 64 and N = 8 as the constrained-search baseline.For the large dataset, effectiveness continued to improve with model depth, so only a global optimum for demb could be obtained.Source: §5.4.1 · p. 8
5.4.2. APPLICATION 2: EXPLORING SCALING LAW POTENTIAL AMONG FRAMEWORK
- Across HSTU, LLaMA2, and SASRec fits, both scaling coefficients were positive, and model performance aligned with the magnitude trend of the fitted exponents.Experiments used ML-1m and float32 and bfloat16 precision settings.Source: §5.4.2 · pp. 8–9, Figure 5 · p. 9
6. Conclusion
The paper addresses the gap between loss and actual performance in sequential recommendation (SR), as well as the effects of data redundancy. It introduces a Performance Law and Approximate Entropy (ApEn) to model performance with attention to data quality and support predictions across dataset scales and model sizes.
6. Conclusion
- The paper addresses the discrepancy between model loss and actual SR performance, along with the adverse effects of data redundancy on model efficacy.Source: §6 · p. 9
- It introduces a Performance Law that fits HR and NDCG for transformer-based SR models and uses ApEn to assess data quality over sheer quantity.Source: §6 · p. 9
- The framework supports performance predictions across varying dataset scales and model sizes and helps balance computational resources toward optimal results.Source: §6 · p. 9