PaperlayerView exact revision on arXiv

Source-linked AI summary

Dataset Regeneration for Sequential Recommendation

Mingjia Yin, Hao Wang, Wei Guo, Yong Liu, Suojuan Zhang, Sirui Zhao, Defu Lian, Enhong Chen

arXiv:2405.17795v3cs.IR

TL;DR

Model-centric sequential recommenders are developed on fixed datasets, often overlooking data quality issues. This paper proposes DR4SR to regenerate training data and DR4SR+ to personalize it for target models. The authors report cross-architecture generalizability and significant improvements across four datasets when integrated with model-centric methods.

  • Problem

    Model-centric sequential-recommendation methods use fixed datasets and may overlook inherent data quality issues.

    Source: §1 · p. 1

  • Method

    DR4SR regenerates data to represent item-transition patterns, while DR4SR+ personalizes regenerated patterns for target models.

    Source: §1 · p. 2

  • Results

    Experiments report cross-architecture generalizability and significant performance improvements across four widely adopted datasets when DR4SR and DR4SR+ are integrated with model-centric methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings underscore the complementary relationship between data-centric and model-centric approaches to sequential recommendation.

    Source: §1 · p. 2, §6 · p. 9

    This study primarily regenerates sequences; extending regeneration to graphs and augmented data is left for future work.

    Source: §5.4.3 · p. 9

Abstract

from arXiv · show

The sequential recommender (SR) system is a crucial component of modern recommender systems, as it aims to capture the evolving preferences of users. Significant efforts have been made to enhance the capabilities of SR systems. These methods typically follow the model-centric paradigm, which involves developing effective models based on fixed datasets. However, this approach often overlooks potential quality issues and flaws inherent in the data. Driven by the potential of data-centric AI, we propose a novel data-centric paradigm for developing an ideal training dataset using a model-agnostic dataset regeneration framework called DR4SR. This framework enables the regeneration of a dataset with exceptional cross-architecture generalizability. Additionally, we introduce the DR4SR+ framework, which incorporates a model-aware dataset personalizer to tailor the regenerated dataset specifically for a target model. To demonstrate the effectiveness of the data-centric paradigm, we integrate our framework with various model-centric methods and observe significant performance improvements across four widely adopted datasets. Furthermore, we conduct in-depth analyses to explore the potential of the data-centric paradigm and provide valuable insights. The code can be found at https://github.com/USTC-StarTeam/DR4SR.

1 INTRODUCTION

Prior sequential-recommendation work mainly improves models on fixed datasets, leaving data quality and training-data development comparatively underexplored. DR4SR regenerates data to expose item-transition patterns, while DR4SR+ personalizes that data for target models; experiments report cross-architecture generalizability and complementarity with model-centric methods.

1 INTRODUCTION

  • The paper frames training-data development as an underexplored data-centric problem for sequential recommendation, where data issues can be overlooked by model-centric methods.The authors describe this as the first investigation of the problem in sequential recommendation.Source: §1 · p. 1
  • The paper distinguishes dataset regeneration from denoising and distillation: its objective is to develop novel data content and formats for improved model training.The cited discussion characterizes denoising as only one part of training-data development, while distillation prioritizes training efficiency.Source: §1 · p. 2
  • DR4SR regenerates the original dataset into informative, generalizable data using pre-training, a diversity-promoted regenerator, and hybrid inference.Its goal is to model a one-to-many mapping from original sequences to item-transition patterns, without adding external information.Source: §1 · p. 2
  • Across four widely adopted datasets, experiments report cross-architecture generalizability and complementarity between DR4SR and model-centric methods.The introduction reports substantial improvements when the frameworks are integrated with model-centric methods.Source: §1 · p. 2
  • DR4SR+ adds a model-aware personalizer that scores regenerated patterns for a target model and uses implicit differentiation in bi-level optimization.The personalization step addresses the possibility that model-agnostic regenerated data may be sub-optimal for a particular target.Source: §1 · p. 2

2 RELATED WORKS

Prior sequential-recommendation research has advanced model design, training strategies, and self-supervised learning, while denoising targets noisy information in interaction data. Data-centric recommendation work includes generation and distillation, but the cited approaches often prioritize privacy or efficiency rather than recommendation performance.

2.1 Sequential Recommendation

  • Sequential recommendation research has developed CNN, RNN, GNN, and Transformer models, alongside improved training strategies and self-supervised tasks.The passage cites model families [43, 61], [8, 11, 26], [53, 57, 60], [20, 42], training strategies [30, 37], and self-supervised work [3, 39, 59, 81].Source: §2.1 · pp. 2–3
  • Denoising methods filter noisy sequence information through representation-space architectures or explicit item filtering, but do not generate diverse, novel samples.The discussion says denoising alone falls short of producing an optimal dataset because it focuses on removing information [1].Source: §2.1 · p. 3

2.2 Data-Centric AI in Recommender System

  • Data-centric recommendation work includes privacy-preserving datasets, adaptive graph generation, long-tail data generation, and dataset distillation.The cited distillation approaches aim to produce condensed datasets that let trained models attain comparable performance [69].Source: §2.2 · p. 3
  • The paper distinguishes its performance-enhancement objective from prior generated-dataset methods that primarily prioritize privacy or training efficiency.The discussion names UPC-SDG [31] and distillation methods [40, 46, 54, 55, 69] in this contrast.Source: §2.2 · p. 3

3 PROBLEM DEFINITION

Sequential recommendation predicts users’ next items from chronologically ordered interaction sequences. This paper frames training data development as constructing an informative, generalizable dataset for learning user preferences, rather than improving a model on fixed data.

3 PROBLEM DEFINITION

  • Sequential recommendation predicts each user’s next item from their chronologically ordered interaction sequence.Sequences contain items interacted with over time, up to a maximum length.Source: §3 · p. 3
  • Training data development seeks a new informative, generalizable dataset on which a target model can learn user preferences more easily.Traditional methods instead learn a target model from the original dataset.Source: §3 · p. 3
  • Dataset regeneration learns a model-independent regenerator that maps each original sequence to K informative patterns, which together form the new dataset.The paper presents model-aware personalization as a further step for tailoring this regenerated dataset to a target recommendation model.Source: §3 · p. 3

4 METHODOLOGY

DR4SR regenerates sequences into informative patterns through pre-training, a diversity-promoted regenerator, and hybrid inference. DR4SR+ adds a target-model personalizer, optimized through a bilevel objective, to tailor the regenerated data.

4.1 Overall Framework

  • DR4SR regenerates the original dataset into an informative, generalizable dataset, while DR4SR+ tailors it to a particular target model.The framework comprises pre-training, regeneration inference, and model-aware personalization stages.Source: §4.1 · pp. 3–4, Figure 2 · p. 4

4.2 Model-agnostic Dataset Regeneration

  • DR4SR uses self-supervised pre-training, a diversity-promoted regenerator, and hybrid inference because the original dataset lacks supervision for learning regeneration.The pipeline is presented as model-agnostic dataset regeneration.Source: §4.2 · p. 4

4.2.1 Construction of Pre-training Task.

  • Pre-training teaches the regenerator to produce patterns extracted from original sequences using rule-based methods.A sliding window of size α counts patterns, and those occurring more than threshold β are selected; sequence-pattern pairs form training instances.Source: §4.2.1 · p. 4

4.2.2 Diversity-promoted Regenerator.

  • A diversity promoter addresses conflicting pattern targets by supplying distinct sequence memories for different patterns during decoding.The regenerator uses an encoder and decoder; it projects sequence memory into K spaces and uses target-pattern information to guide memory selection.Source: §4.2.2 · pp. 4–5

4.2.3 Dataset Regeneration with Hybrid Inference Strategy.

  • Hybrid inference generates K patterns per sequence by decoding with each memory and mixing restrictive selection from input items with unrestricted generation.The generative mode can explore patterns not explicitly present in the original dataset; probability γ selects that mode and 1 − γ selects restrictive decoding.Source: §4.2.3 · p. 5

4.3 Model-aware Dataset Regeneration

  • DR4SR+ adds a dataset personalizer because model-independent regeneration may produce data sub-optimal for a particular target model.The personalizer scores regenerated samples and is optimized using implicit differentiation.Source: §4.3 · p. 5

4.3.1 Dataset personalizer.

  • The personalizer scores each regenerated pattern from its target-model representation, allowing pattern-level evaluation.It is an MLP whose Gumbel-noise-based score is used to weight training losses, without requiring modifications to target models.Source: §4.3.1 · p. 5

4.3.2 Efficient optimization of dataset personalizer.

  • The personalizer is optimized with a bilevel objective: train the target model on weighted data, then update the personalizer against unweighted validation loss.The implementation performs an upper-level step after T lower-level optimization rounds and uses a vector-Jacobi product to compute the gradient without substantial time overhead.Source: §4.3.2 · p. 6

5 EXPERIMENTAL EVALUATION

Across datasets and target models, DR4SR improves recommendation performance over the original-data backbone, and DR4SR+ adds further improvement. Ablations and analyses examine the framework’s components, personalization, data forms, and hyperparameters.

5.1 Experimental Setup

  • Experiments use Beauty, Sports and Outdoors, Toys and Games, and Yelp, retaining users and items with at least five interactions.Target models span RNN, Transformer, denoising, GNN, and contrastive-learning categories; evaluation uses leave-one-out next-item recommendation with Recall and NDCG metrics.Source: §5.1.1 · p. 6, §5.1.2 · p. 6, §5.1.3 · p. 6

5.2 Overall Performance

  • DR4SR improves performance across datasets and target models over the backbone, while DR4SR+ provides additional improvement.The authors also report that data-centric and model-centric paradigms are complementary; Table 3 presents overall performance.Source: §5.2 · p. 7, Table 3 · p. 8

5.3 Ablation Study

  • Ablations report performance loss without diversity-promoted regeneration, with predefined patterns alone, and with end-to-end personalizer optimization.The authors associate these outcomes with the one-to-many mapping, infrequent patterns, and the risk of a trivial all-zero personalizer solution.Source: §5.3 · p. 7, Table 4 · p. 8

5.4 Advanced Study

  • Implicit differentiation for personalizer training adds no significant time or space overhead, and score visualizations show target-model differences in pattern preferences.GRU4Rec has greater score variance than the other visualized models; Figure 4 reports variances of 0.0280, 0.0244, 0.0241, 0.0248, and 0.0247.Source: §5.4.1 · p. 7, §5.4.2 · p. 7, Table 5 · p. 8

5.4.3 Analysis of data forms that should be regenerated

  • Graph construction results are mixed, while regenerated sequences improve CL4SRec augmentation on the first two datasets but show no significant improvement on the last two.The authors conclude that some advanced data formats may still need to be constructed from original data, and this study focuses on sequence regeneration.Source: §5.4.3 · pp. 7–9, Figure 5 · p. 8

5.5.1 The diversity factor K.

  • Increasing diversity factor K improves performance until a threshold, after which further increases bring no significant change.The authors attribute the trend to reduced training confusion and increased regenerated-data diversity, with the plateau indicating captured patterns.Source: §5.5.1 · p. 9, Figure 6 · p. 9

5.5.2 The generative decoding probability

  • Performance first rises and then falls as generative decoding probability γ increases, underscoring the need to balance exploration and exploitation.The authors attribute low γ to insufficient exploration of higher-order information and high γ to underuse of existing information.Source: §5.5.2 · p. 9, Figure 7 · p. 9

6 CONCLUSIONS

The paper presents DR4SR as a data-centric framework for developing sequential-recommendation training data, with cross-architecture generalizability and complementary benefits alongside model-centric methods. It extends the framework with DR4SR+ for target-model personalization and identifies broader data formats and language-model guidance as future directions.

6 CONCLUSIONS

  • The conclusion presents DR4SR as a training-data regeneration framework with cross-architecture generalizability and complementarity with model-centric methods.DR4SR+ extends it with personalized adaptation of regenerated datasets to target models.Source: §6 · p. 9
  • Future work will explore regenerating sequences, graphs, and augmented data, and using large language models to guide regeneration.The proposed language-model integration aims to generate data retaining collaborative and semantic information.Source: §6 · p. 9

A.1 Pseudo Code of DR4SR and DR4SR+

The section identifies Algorithm 1 as the detailed pipeline for DR4SR and Algorithm 2 as the detailed pipeline for DR4SR+.

A.1 Pseudo Code of DR4SR and DR4SR+

  • Algorithm 1 details the DR4SR pipeline, while Algorithm 2 details the DR4SR+ pipeline.Source: §A.1 · p. 11

A.2 Detailed Derivations of Equation 10

The derivation computes the development-loss gradient through the optimal training parameters using the chain rule. It obtains the training-parameter optimum’s implicit gradient through a truncated Neumann series.

A.2 Detailed Derivations of Equation 10

  • The chain rule expresses the implicit development-loss gradient as a product involving the gradient with respect to the optimal training parameters and their dependence on ϕ.The derivation then uses the optimality condition for the training loss to obtain that dependence.Source: §A.2 · p. 11
  • Automatic differentiation computes the development-loss gradient with respect to the optimal training parameters.Source: §A.2 · p. 11
  • The derivation uses a K-truncated Neumann series [35, 36] to calculate the gradient with respect to ϕ and obtain the final implicit gradient.Source: §A.2 · p. 11

A.3 Computational Complexity

The complexity analysis separates dataset preparation, model pre-training, inference, and personalization, and identifies implicit-gradient computation as a major cost. It reports that DR4SR and DR4SR+ are scalable with acceleration strategies.

A.3 Computational Complexity

  • Dataset scale is characterized by sequence count N, maximum sequence length L, maximum pattern length M, and extracted-pattern count q; K denotes the diversity factor.Empirically, q is of the same order of magnitude as N.Source: §A.3 · p. 12
  • Pre-training data construction with Seq2Pat has complexity O(NMq), while generator pre-training has complexity O(qL^2d).The pattern-mining step can use offline pre-computation or batching.Source: §A.3 · p. 12
  • Inference has complexity O(KNL^2d) because it regenerates K patterns per sequence; efficient autoregressive decoding [58] can accelerate it.Source: §A.3 · p. 12
  • Implicit-gradient computation is a main complexity source, requiring K+2 backward vector-Jacobi products [36], with the truncated Neumann-series number K fixed at 3.Source: §A.3 · p. 12
  • DR4SR and DR4SR+ are reported to be scalable on large datasets when acceleration strategies are used.Source: §A.3 · p. 12

A.4 Detailed Hyper-parameters

The appendix specifies target-model and DR4SR hyperparameters, then reports generalization results on Amazon-book and Gowalla. The reported results support generalization to large-scale datasets or datasets with different distributions.

A.4 Detailed Hyper-parameters

  • Target-model settings include two-layer SASRec and FMLP configurations, while GRU4Rec and GCE-GNN use searched layer counts; GCE-GNN’s encoders are replaced with LightGCN [9] and SASRec [20].CL4SRec uses contrastive-loss weight 0.1 and crop/mask/reorder rates 0.2/0.7/0.2; other hyperparameters match SASRec.Source: §A.4 · p. 12
  • DR4SR uses sliding-window size 5 for Beauty and 10 for other datasets, threshold 2, diversity number 5, and generative-decoding probability 0.1.Source: §A.4 · p. 12
  • For DR4SR+, T_lower is set to 30, and SGD is the upper optimizer with learning rate and weight decay searched over [1e-3, 1e-2, 1e-1].Source: §A.4 · pp. 12–13

A.5 Generalization to More Datasets

  • Amazon-book and Gowalla results are reported as evidence that DR4SR generalizes to large-scale datasets or datasets with different distributions.Source: §A.5 · p. 13
Loading 2405.17795v3…