PaperlayerView exact revision on arXiv

Source-linked AI summary

Principled Synthetic Data Enables the First Scaling Laws for LLMs in Recommendation

Benyu Zhang, Qiang Zhang, Jianpeng Cheng, Hong-You Chen, Qifei Wang, Wei Sun, Shen Li, Jia Li, Jiahao Wu, Qunshu Zhang, Neeraj Bhatia, Xiangjun Fan, Hong Yan

arXiv:2602.07298v3cs.IRcs.AI

TL;DR

Predictable scaling laws have been missing for LLM-based recommendation, amid concerns that raw interaction data are noisy and biased. The paper introduces a layered synthetic-data curriculum and reports stronger ranking performance, robust power-law scaling, and actionable data-mixing guidance, while production end-to-end validation remains underway.

  • Problem

    LLM recommendation lacks predictable scaling laws, and prior continual pre-training relies on noisy, biased, and incomplete interaction data.

  • Method

    The paper creates a layered synthetic-data curriculum using item-text alignment, collaborative filtering, and graph-based random walks for position-debiased user histories.

  • Results

    Synthetic-data-trained sequential models outperform real-data-trained models, while LLMs show robust power-law scaling across modalities and asymmetric CF-to-UIH transfer.

  • Takeaways & Limitations

    The findings provide a quantitative roadmap for recommendation data curation, resource allocation, and data mixing.

  • Takeaways & Limitations

    Production evaluation of the CPT-trained LLM on end-to-end recommendation metrics in a live system is underway.

Abstract

from arXiv · show

Large Language Models (LLMs) represent a promising frontier for recommender systems, yet their development has been impeded by the absence of predictable scaling laws, which are crucial for guiding research and optimizing resource allocation. We hypothesize that this may be attributed to the inherent noise, bias, and incompleteness of raw user interaction data in prior continual pre-training (CPT) efforts. This paper introduces a novel, layered framework for generating high-quality synthetic data that circumvents such issues by creating a curated, pedagogical curriculum for the LLM. We provide powerful, direct evidence for the utility of our curriculum by showing that standard sequential models trained on our principled synthetic data significantly outperform ($+130\%$ on recall@100 for SasRec) models trained on real data in downstream ranking tasks, demonstrating its superiority for learning generalizable user preference patterns. Building on this, we empirically demonstrate, for the first time, robust power-law scaling for an LLM that is continually pre-trained on our high-quality, recommendation-specific data. Our experiments reveal consistent and predictable perplexity reduction across multiple synthetic data modalities. These findings establish a foundational methodology for reliable scaling LLM capabilities in the recommendation domain, thereby shifting the research focus from mitigating data deficiencies to leveraging high-quality, structured information.

1 Introduction

The paper argues that noisy, biased interaction logs hinder predictable recommendation scaling and introduces a layered synthetic-data curriculum to address this problem. It reports stronger downstream ranking performance, robust power-law scaling, and asymmetric transfer between data modalities.

  • Raw interaction logs contain noise, sparsity, and position, popularity, and exposure biases that complicate predictable scaling.
  • A layered curriculum transforms flawed logs into semantic, collaborative, and position-debiased behavioral data.Layer 1 provides item-text alignment and collaborative filtering data; Layer 2 generates position-debiased user histories.
  • Standard sequential models trained on synthetic data outperform models trained on real data across Recall@10, Recall@100, and Recall@1000.The comparison includes GRU4Rec, NARM, STAMP, and SASRec.
  • The resulting scaling hierarchy provides a quantitative roadmap for data curation and resource allocation.
  • 0.6B to 8B parameters and 163B tokens yield robust power-law scaling across seven recommendation evaluation domains.User interaction history has the strongest scaling, followed by collaborative filtering and item-text alignment.
  • 31% improvement follows when collaborative filtering data is added to user interaction histories, while reverse transfer does not hold.The reported asymptotic UIH loss changes from 0.95 to 0.66.

2 Related Work

Related work studies scaling, synthetic data, debiasing, and graph-based recommendation through model, system, distillation, or augmentation strategies. This paper instead adopts a data-centric curriculum that purifies recommendation signals and enables standard LLM training to scale predictably.

  • 2.1 Scaling Laws in Recommender Systems: Scaling laws relate loss to model size, dataset size, and compute, while Chinchilla scaling requires model and dataset sizes to grow together under fixed compute.Transformer compute can be estimated as C = 6ND.
  • 2.1 Scaling Laws in Recommender Systems: Predictable scaling laws had not yet been established for recommendation models.
  • 2.1 Scaling Laws in Recommender Systems: Prior recommendation approaches are model-centric, system-centric, or distillation-centric while relying primarily on raw interaction logs.
  • 2.2 Principled Synthetic Data Generation for Recommendation: Traditional synthetic-data methods augment or replicate source distributions, whereas this framework purifies signals through a debiased collaborative-filtering graph and generates new sequences with random walks.
  • 2.2 Principled Synthetic Data Generation for Recommendation: The framework operates at the curriculum level, structuring learning across semantics, collaborative structure, and sequential behavior.
  • 2.3 Data-Centric Debiasing: Unlike model-side debiasing methods, the data-centric approach engineers position-debiased data for standard LLM architectures.
  • 2.4 Graph-Based Methods for Recommendation: Graph-based methods provide related interaction modeling, but this framework differs by constructing a pedagogical data curriculum rather than modifying model architectures.

3 The Data Quality Bottleneck in Recommendation CPT

LLM adaptation for recommendation faces efficiency, cold-start, domain-gap, and data-quality challenges. Raw interaction logs encode systemic biases that can be amplified during continual pre-training, while insufficient data can produce sub-scaling.

  • Large models can introduce recommendation latency, and limited histories create cold-start difficulties for new users or items.
  • The domain gap requires connecting general-purpose LLM knowledge with idiosyncratic user-behavior patterns, making adaptation-data quality central.
  • Raw interaction logs are distorted by system-generated effects rather than reflecting pure user preference, including categorized systemic biases.
  • Continual pre-training can make LLMs internalize dominant statistical regularities in the data, including its biases.
  • Insufficient training data prevented PLUM’s MoE-3B model from showing clear superiority over the smaller MoE-900M model, illustrating sub-scaling.
  • When data lacks diversity or contains redundancy, additional data yields diminishing marginal benefits and models deviate from expected power-law trajectories.
  • Biased continual pre-training can amplify recommendation skew through a feedback cycle in which newly generated logs become more contaminated.

4 A Layered Framework for High-Fidelity Synthetic Data

The framework constructs a layered, pedagogical synthetic-data curriculum that grounds item and collaborative knowledge before generating realistic, privacy-preserving interaction sequences. Its graph-based pipeline uses collaborative relationships and biased random walks to separate preference signals from system artifacts.

  • The framework replaces raw-data processing with a curated curriculum designed to teach recommendation concepts progressively and with high fidelity.
  • Layer 1: Layer 1 teaches item meaning and relationships through item-text alignment and collaborative-filtering signals.
  • Item-Text Alignment: Item-text alignment maps semantic item tokens to descriptive text, grounding representations in language rather than only ID co-occurrences.
  • Collaborative Filtering: Collaborative-filtering data converts mined association rules and item-to-item relationships into explicit natural-language learning signals.
  • Layer 2: Layer 2 generates clean, diverse, privacy-preserving interaction sequences intended to address missing positives and position bias.
  • Synthetic UIH: The CF-to-LLM pipeline builds a weighted item graph and simulates journeys with 2nd-order biased random walks conditioned on current and previous nodes.
  • The layered process extracts co-occurrence signals from aggregated sessions, reducing susceptibility to individual position and presentation bias.

5 Empirical Validation of Synthetic Data Quality

The evaluation examines fidelity, ranking utility, and privacy. Synthetic-data-trained sequential recommenders outperform real-data-trained baselines across models and cutoffs, while graph-generated sequences remain decoupled from individual histories.

  • The framework evaluates synthetic data along fidelity, utility, and privacy dimensions.
  • Fidelity: Fidelity compares item-popularity and sequence-length distributions between Synthetic UIH and original Merrec logs.
  • Utility: TSTR compares synthetic-trained, real-tested models with a TRTR baseline using a real test set restricted to shared-vocabulary items.
  • Utility: Synthetic-trained GRU4Rec, NARM, STAMP, and SASRec models achieve higher Recall@K than real-trained models at @10, @100, and @1K.
  • Utility: The ranking results support synthetic data as a superior training resource for learning generalizable user-preference co-occurrence patterns.
  • Privacy: Synthetic UIH sequences derive from an aggregated item-relationship graph rather than individual histories, mitigating privacy risks tied to raw logs.

6 Principled Scaling Laws for Recommendation

Continual pre-training uses Qwen3 models from 0.6B to 8B parameters on a mixed general and recommendation dataset totaling 163B tokens. Across seven domains, perplexity follows consistent power-law trends, with recommendation modalities—especially UIH—showing stronger data scaling than general or item-text data, while compute-optimal allocation differs by domain.

  • Experimental Configuration: The training dataset mixes general-domain and recommendation data at a 50% and 50% ratio to avoid catastrophic forgetting.Recommendation data comprises three synthetic data types from two layers.
  • Per-Model Scaling Laws: Across General Domain, Item-Text, CF, and UIH variants, empirical perplexity points follow consistent power-law trends as training tokens increase.The evaluation covers seven modalities, including seen and unseen CF settings and two UIH settings.
  • Per-Model Scaling Laws: UIH has the strongest data scaling (α ≈0.45–0.59), followed by CF (α ≈0.28–0.36), Item-Text (α ≈0.13–0.21), and General Domain (α ≈0.02–0.03).General-domain near-saturation is attributed to the pretrained Qwen3 checkpoint already containing substantial knowledge.
  • Per-Model Scaling Laws: Larger models exhibit higher α for CF and UIH, while L∞ remains nearly constant across model sizes in those domains.This indicates that recommendation-domain convergence is limited more by data coverage than model capacity.
  • Joint Scaling Law: Compute allocation is model-size dominant for General and Item-Text, data dominant for UIH, and intermediate for CF.The joint fit reports General/Item-Text with high α and low β, UIH with low α and high β, and CF between them.
  • Joint Scaling Law: Halving the data-dependent loss term requires approximately 1.8×10^6 times more data for General but only approximately 13 times more data for UIH.The contrast quantifies why data collection is substantially more valuable for UIH than General under the reported scaling laws.
  • Joint Scaling Law: The joint additive formulation does not fully capture the model-data interaction indicated by increasing αdata with model size for CF and UIH.The paper identifies this interaction as an avenue for future investigation.

7 Scaling Analysis and Future Directions

The ablation studies examine how synthetic-data layers and mixture ratios affect recommendation scaling, transfer, and overfitting across model sizes. They find complementary domains, asymmetric CF-to-UIH transfer, and repetition-dependent degradation.

  • Ablation Study: The Synergistic Effect of Layered Data: The ablation compares UIH-only, CF+UIH, and complete Item-Text+CF+UIH curricula while holding experiments to 163B tokens and 50% general text.The study fits scaling laws and evaluates perplexity across multiple domain-specific hold-out sets.
  • Ablation Study: The Synergistic Effect of Layered Data: CF+UIH achieves UIH asymptotic perplexity L∞= 0.66 versus L∞= 0.95 for UIH-only, indicating beneficial transfer from collaborative filtering to sequential behavior modeling.The reverse transfer is negligible, with UIH-only yielding α ≈0 on CF tasks.
  • Ablation Study: The Synergistic Effect of Layered Data: L∞= 0.35 with CF data versus L∞= 1.30 without CF, while item-text exclusion raises item-text loss from approximately 1.2 to approximately 3.These results show that domain-specific data remains important even when cross-domain transfer is available.
  • Ablation Study: The Synergistic Effect of Layered Data: Including item-text data raises UIH loss from L∞= 0.66 for CF+UIH to L∞= 0.76 for Item+CF+UIH, creating a modest multi-domain trade-off.The item-text layer prevents catastrophic forgetting of item semantics.
  • Ablation Study: Data Mixtures: At 2% UIH mixture, evaluation perplexity increases after about 16 repeats, while training loss continues decreasing, signaling overfitting from repeated reduced data.At 15% mixture, overfitting begins at 20B tokens; at 5%, it begins at 80B tokens.
  • Ablation Study: Data Mixtures: At 15% UIH mixture, evaluation perplexity rises at 20B tokens, versus 160B tokens at 2%, independently of model scale but more strongly for larger models.The onset therefore depends on mixture ratio rather than parameter count.
  • Scaling Across Model Size and Data: Recommendation-specific scaling differs by domain: CF saturates at 4B parameters with 1.47 PPL, while UIH on unseen items shows inverse scaling, with 0.6B reaching 2.01 PPL versus 2.10 PPL for 8B.The models were trained on 163.84B tokens with a mixed-domain curriculum.

8 Case Studies

The case studies illustrate recommendation inference from user interaction histories using semantic IDs, beam search, and similarity retrieval. The examples emphasize relevant, diverse outputs and semantic item representations.

  • Recommendation given UIH: The model recommends an item from a UIH, with its response matching the ground truth except for the color.The example presents both semantic-ID format and decoded natural language.
  • Recommendation given UIH: Beam search produces recommendations that are both relevant and diverse in the illustrated inference example.The figure shows semantic-ID input/output alongside decoded natural language.
  • Recommendation given a Random UIH: Given a random UIH, the model still recommends items relevant to the presented items, with greater diversity because the history itself is random.The figure again displays semantic-ID outputs and decoded natural language.
  • Semantic-ID Similarity Search: Semantic IDs support retrieving the top-10 most similar items for a generated item, illustrating their role in representing item semantics.The example uses similarity search based on semantic IDs.

9 Discussion and Conclusion

The discussion argues that predictable recommendation scaling is enabled by high-quality synthetic data rather than guaranteed by the domain itself. It connects scaling exponents and cross-domain transfer to practical data-mixing guidance while noting limits of perplexity-based validation.

  • Discussion and Conclusion: The paper attributes the prior absence of predictable scaling laws to pathologically flawed raw user data rather than an inherent barrier in recommendation.The conclusion frames data quality as the foundational issue addressed by the work.
  • Discussion and Conclusion: UIH has higher learning efficiency than Item-Text, with α=0.45–0.59 versus α=0.15, while CF accelerates UIH learning by 31%.These results provide guidance for mixing recommendation data modalities.
  • Discussion and Conclusion: The layered synthetic-data framework enables robust, predictable power-law scaling across recommendation models and datasets.The paper reports stable scaling for Gemma 3 and Qwen3 on a proprietary social-media dataset, with RMSE < 0.035.
  • Cross-Model and Cross-Dataset Generalization: The social-media extension preserves the hierarchy α: UIH > Alignment > General and reports stronger UIH data scaling, β=0.894 versus β=0.272 on MerRec.The inverse relationship between model-size and data-scaling exponents is also preserved.
  • Perplexity vs. Downstream Metrics: Perplexity is supported as practically relevant by TSTR ranking results and correlations with Recall@K and NDCG, but live end-to-end production evaluation remains underway.The paper treats perplexity as analogous to loss-based scaling metrics used in Chinchilla.
  • Practical Implications: The established scaling laws are intended to replace heuristic planning with forecasts of computational budgets and data volumes for target performance.This is presented as a practical implication for recommendation-system development.

Impact Statement

The impact statement highlights privacy and potential fairness benefits from synthetic recommendation data while cautioning that downstream choices can reintroduce bias. It also describes methodological safeguards and risks requiring further auditing.

  • Positive Societal Impacts: Synthetic user histories are derived from aggregated item-to-item graphs rather than individual user traces, providing inherent privacy benefits.This construction decouples training data from real user behavior.
  • Positive Societal Impacts: Position-debiased synthetic curricula produce more uniform item exposure, with Gini 0.64 versus >0.95 in real logs, and may surface long-tail content.The paper presents this as a potential fairness benefit rather than a guaranteed downstream outcome.
  • Potential Risks and Mitigations: Downstream applications may introduce new biases through source-data selection or graph construction, so practitioners should audit synthetic-data fidelity and fairness.The paper also notes that more effective recommendations could raise digital-well-being concerns through increased engagement.
  • Synthetic UIH Generation: The UIH generator uses a Node2Vec-based random-walk sweep spanning 14 configurations, including DeepWalk-style and BFS-like strategies.The sweep varies walk termination and path-confidence filtering parameters.
  • Semantic Item Representation: SAE was selected over in-domain RQ-VAE and RQ-kmeans for semantic item representation based on ablation experiments.The representation serves as the LLM’s fundamental vocabulary.

B.1 Candidate Methodologies

The study compares SAE, RQ-VAE, and RQ-kmeans tokenization approaches before selecting SAE, then evaluates the synthetic-data framework’s bias mitigation and experimental isolation. SAE outperformed RQ-kmeans across domains and steps, while the framework removes positional and temporal-ordering artifacts but only partially mitigates popularity bias.

  • Candidate tokenization methods: SAE extracts top-k activated concepts from item-text embeddings using a pre-trained sparse autoencoder, whereas RQ-VAE quantizes target-domain embeddings with a residual codebook.SAE relies on an internal multimodal encoder; RQ-VAE is trained from scratch on target item descriptions.
  • Candidate tokenization methods: RQ-kmeans reached a 5.33% collision rate with six codebook layers and 256 codes per layer, while RQ-VAE reached 50.54%.The sweep used Qwen-0.6B embeddings for 64.1M items, with beam size 5, progressive search, and uniform sampling.
  • Selected methodology: SAE was selected after ablation because it consistently outperformed RQ-kmeans across all domains and steps on 4B-model scaling-law comparisons.The compared settings were otherwise identical.
  • Bias mitigation: The framework eliminates positional and temporal-ordering bias by generating random walks from graph structure rather than rank, presentation order, or real-session chronology.Each walk step depends on graph structure and p, q parameters.
  • Bias mitigation: Popularity bias remains partially mitigated because co-occurrence graph edge weights give popular items higher visitation probabilities during random walks.Synthetic UIH has Gini coefficients of 0.635–0.732 versus near-1.0 in real logs, indicating reduced but nonzero concentration.
  • Experimental controls: The experiments isolate training information by constructing the CF graph and SAE tokenizer from training splits, preventing UIH overlap and excluding test-derived graph structure outside evaluation.Test items may be fully or partially held out depending on the evaluation split.
Loading 2602.07298v3…