PaperlayerView exact revision on arXiv

Source-linked AI summary

Tenrec: A Large-scale Multipurpose Benchmark Dataset for Recommender Systems

Guanghu Yuan, Fajie Yuan, Yudong Li, Beibei Kong, Shujie Li, Lei Chen, Min Yang, Chenyun Yu, Bo Hu, Zang Li, Yu Xu, Xiaohu Qie

arXiv:2210.10629v3cs.IR

TL;DR

Existing recommender-system benchmarks can be small-scale or limited in feedback types, complicating evaluation on real-world recommendation problems. Tenrec combines multiple scenarios, feedback types, and additional features, and the paper examines it across ten recommendation tasks. In session-based recommendation, three unidirectional models perform similarly, with NextItNet slightly better than SASRec and GRU4Rec.

  • Problem

    Existing recommender-system evaluations often rely on non-benchmark or small-scale datasets, and public datasets commonly provide limited feedback types or recommendation scenarios.

    Source: §1 · p. 1

  • Method

    Tenrec combines overlapping users and items across four scenarios, varied positive and true negative feedback, and features beyond user and item IDs; the paper examines it on ten recommendation tasks.

    Source: §1 · p. 2

  • Results

    Across ten tasks Tenrec is used to examine recommendation models; in session-based results, three unidirectional models perform similarly, with NextItNet slightly better than SASRec and GRU4Rec.

    Source: §1 · p. 2, §3.2 · p. 6

  • Takeaways & Limitations

    Tenrec is presented as a dataset for studying a broad range of recommendation tasks and as a potential standardized benchmark for evaluating progress.

    Source: §4 · p. 9

    The paper has not explored Tenrec's full application potential and plans further scenario coverage and future versions with item modality information.

    Source: §4 · p. 10

Abstract

from arXiv · show

Existing benchmark datasets for recommender systems (RS) either are created at a small scale or involve very limited forms of user feedback. RS models evaluated on such datasets often lack practical values for large-scale real-world applications. In this paper, we describe Tenrec, a novel and publicly available data collection for RS that records various user feedback from four different recommendation scenarios. To be specific, Tenrec has the following five characteristics: (1) it is large-scale, containing around 5 million users and 140 million interactions; (2) it has not only positive user feedback, but also true negative feedback (vs. one-class recommendation); (3) it contains overlapped users and items across four different scenarios; (4) it contains various types of user positive feedback, in forms of clicks, likes, shares, and follows, etc; (5) it contains additional features beyond the user IDs and item IDs. We verify Tenrec on ten diverse recommendation tasks by running several classical baseline models per task. Tenrec has the potential to become a useful benchmark dataset for a majority of popular recommendation tasks.

1 Introduction

Recommender-system research faces reproducibility and credibility problems because many evaluations use non-benchmark or small-scale datasets, while existing datasets often offer limited feedback types or scenarios. Tenrec is proposed as a large-scale, multipurpose dataset with overlapping users and items, diverse feedback, and additional features, and is examined across ten recommendation tasks.

1 Introduction

  • Existing recommender-system evaluations often use non-benchmark or small-scale datasets, contributing to reproducibility and credibility problems [37].Prior work also questioned reported advances from dataset and experimental-setup perspectives [9] [20] [1].Source: §1 · pp. 1–2
  • Many popular public datasets contain only one feedback type or one recommendation scenario, limiting the scope of real-world recommender-system research.The passage names MovieLens, Netflix, Yelp, Amazon, MIND, Last.fm, and Yahoo! Music as examples.Source: §1 · p. 2
  • Tenrec combines overlapping users and items across four scenarios, multiple positive feedback types, true negative feedback, and features beyond user and item IDs.These properties support study of cross-domain recommendation, transfer learning, multi-task learning, CTR prediction, and context/content-based recommendation.Source: §1 · p. 2
  • Tenrec is evaluated on ten recommendation tasks, and the authors release its datasets and code to promote reproducibility and advance research.The tasks include CTR prediction, session-based and multi-task recommendation, cross-domain recommendation, user-profile prediction, cold-start recommendation, lifelong representation learning, model compression, and training and inference speedup.Source: §1 · p. 2

2 Dataset Description

Tenrec combines data from four feed scenarios, with user behaviors collected as positive and negative feedback and with user and item features. Its cross-scenario overlaps and varied feedback support transfer, cross-domain, multi-task, and CTR recommendation studies.

2 Dataset Description

  • Tenrec draws from four scenarios—QK-video, QK-article, QB-video, and QB-article—across Tencent's QQ KAN and QQ BOW feed platforms.QK-video is treated as the main scene and the other three as secondary scenes for cross-domain and transfer-learning tasks.Source: §2 · pp. 2–3
  • The QK-video collection starts from around 5.02 million users and yields around 142 million clicks, 10 million likes, 1 million shares, and 0.86 million follows alongside 3.75 million videos.The collection includes exposed items with no user action as negative feedback; the Table 1 caption defines exposure as including both positive and negative feedback.Source: §2 · p. 3, Table 1 · p. 3
  • QK-video item popularity follows a long-tail distribution, and sessions of length 0–20 account for 53% of sessions.Similar distributions are reported for the other three datasets but omitted.Source: §2 · p. 3, Figure 2 · p. 4
  • QK-video shares 268,207 users with QK-article, 3,261 with QB-video, and 58 with QB-article; 78,482 videos overlap between QK-video and QB-video.Overlapped users and items can be associated by unique IDs, supporting transfer-learning and cross-domain recommendation studies.Source: §2 · p. 3
  • Video scenarios include clicks, likes, shares, and follows, while QK-article adds reading and favorite behaviors; exposed but unclicked items provide true negative feedback.The paper notes that likes, shares, and follows often indicate higher preference than clicks, and that true negatives suit CTR prediction.Source: §2 · p. 3
  • Video instances include user and item IDs, feedback actions, category, watching times, gender, and age; article instances add reading, favorite, count, category, score, and reading-time fields.Timestamps were removed, while interaction behaviors are presented in time order; user age is grouped into 10-year bins.Source: §2 · pp. 3–4

3 Experimental Evaluation

Tenrec is evaluated across ten recommendation tasks using popular or state-of-the-art baselines. Results cover prediction, transfer, lifelong learning, compression, and model speedup, with outcomes varying by task.

3.1 CTR Prediction; 3.2 Session-based Recommendation

  • NextItNet slightly outperforms SASRec and GRU4Rec under the same training manner, while unidirectional models beat BERT4Rec; on CTR prediction, NFM performs best and Wide & Deep worst, with around 0.5% disparity.For CTR, the compared metric is AUC; for session-based recommendation, comparisons use HR@20 and NDCG@20.Source: Table 3 · p. 5, §3.1 · p. 5, Table 2 · p. 5

3.3 Multi-task Learning for Recommendation

  • ESMM outperforms MMOE for both click and like prediction, while MMOE does not notably outperform single-objective optimization but can trade off performance across multiple objectives.The MTL experiment uses click and like as its two output objectives.Source: §3.3 · p. 6, Table 4 · p. 6

3.5 User Profile Prediction; 3.6 Cold-start Recommendation

  • Pre-training improves PeterRec and BERT4Rec for user-profile prediction and cold-user recommendation, with pre-trained BERT4Rec outperforming PeterRec in the cold-user setting.The user-profile evaluation reports classification accuracy; the cold-user comparison uses overlapped users across QK-video and QK-article.Source: §3.6 · p. 7, Table 7 · p. 7, §3.5 · p. 7

3.4 Transfer Learning for Recommendation; 3.7 Lifelong User Representation Learning

  • Pre-training improves NextItNet and SASRec top-N results in transfer learning, while Conure-NextItNet raises NDCG@20 from 0.0081 to 0.0095, 0.0160 to 0.0167, and 0.0902 to 0.1074 across lifelong-learning tasks 2–4.The lifelong-learning transfer sequence is QK-video to QK-article to QB-video to QB-article.Source: §3.4 · p. 6, Table 5 · p. 6, §3.7 · p. 8

3.8 Model Compression

  • CpRec compresses NextItNet and SASRec to two thirds of their original sizes with around 2% accuracy drop.The reported result concerns parameter compression for session-based recommendation models.Source: §3.8 · p. 8, Table 9 · p. 8

3.9 Model Training Speedup

  • StackRec reduces training time without lowering recommendation accuracy; Stack64-NextItNet trains 2× faster than standard 16-layer NextItNet.The comparison uses a 64-layer model against the standard 16-layer model.Source: §3.9 · p. 9, Table 10 · p. 9

3.10 Model Inference Speedup

  • SkipRec speeds inference by around 23% for NextItNet and 32% for SASRec while keeping recommendation accuracy on par with the original networks.SkipRec32-NextItNet is also faster than the original 16-layer NextItNet.Source: §3.10 · p. 9, Table 11 · p. 9

4 Conclusions

Tenrec is presented as a large, versatile recommendation dataset spanning real-world scenarios and varied user feedback, with experiments across ten recommendation tasks. The authors also identify unexamined applications and planned dataset extensions.

4 Conclusions

  • Tenrec covers multiple real-world scenarios and varied user feedback, and the paper benchmarks neural models on ten recommendation tasks.The authors release code, datasets, and per-task leaderboards, hoping Tenrec will become a standardized benchmark.Source: §4 · p. 9
  • Video-6M is a very large-scale sequential recommendation dataset for evaluating models with up to 128 layers.The authors report releasing it alongside Tenrec.Source: §4 · p. 9
  • The authors have not explored Tenrec’s full application potential because of space limitations.Planned work includes more scenarios, cross-domain recommendation with overlapped items, feedback-based transfer, negative sampling, and item-modality data for multimodal recommendation.Source: §4 · p. 10

A Dataset Comparison

Tenrec is distinguished from many recommendation datasets by cross-scenario overlap, richer feedback, and true negative examples. The authors contrast its recommendation-system domains with Amazon’s item-category-based domains.

A Dataset Comparison

  • Tenrec’s overlapped users and items across scenarios support studying cross-domain and transfer-learning methods, while varied positive feedback supports multi-task and preference-level transfer learning.Tenrec also includes true negatives for more realistic CTR prediction.Source: A Dataset Comparison · p. 14
  • Unlike Tenrec’s domains, which come from different recommender systems or algorithms, Amazon’s domains are divided by item category, so their suitability for cross-domain recommendation depends on whether recommendation models differ.The authors note that Tenrec-QKA itself includes many article categories.Source: A Dataset Comparison · p. 14

B Supplementary experiment

Supplementary experiments report results for CTR and sequential recommendation on the full QK-video dataset, examine shared historical embeddings, and add mixed cold- and warm-user settings. The appendix also describes top-N baselines and per-task leaderboards.

B Supplementary experiment

  • CTR and sequential recommendation results are reported on the full QK-video dataset with 5 million users, following the main-body experimental setup.The appendix says the main body reports several top-ranked baselines for each task; Tables 13 and 14 cover CTR and SBR, respectively.Source: B Supplementary experiment · p. 14, Table 13 · p. 15, Table 14 · p. 15
  • CTR models with shared historical embeddings generally slightly underperform those with separate historical embeddings, while retaining a similar accuracy ranking.Table 15 reports the shared-embedding experiment on QK-video-1M.Source: B Supplementary experiment · p. 14, Table 15 · p. 15
  • The mixed cold- and warm-user setting holds warm-user behaviors fixed and creates cold users by withholding their latest one to five interactions.Training uses all warm-user behaviors and half of cold-user behaviors; evaluation measures cold-user prediction accuracy with separate validation and test interactions.Source: B Supplementary experiment · pp. 14–15
  • Top-N QB-video baselines use an 8:1:1 per-user split, four negative examples per user, and random or popularity-based negative sampling.The authors caution that stronger negative samplers can improve accuracy, so architecture comparisons should keep loss and sampling settings consistent.Source: B Supplementary experiment · p. 15
  • The authors plan per-task leaderboards for both the full dataset and the 1 million-user version.The passage states this applies to the other tasks.Source: B Supplementary experiment · p. 15

C.1 Motivation

Tenrec was proposed to broaden recommendation research with a dataset spanning multiple scenarios, feedback types, and user and item features. The datasheet describes its composition and collection, notes sampling and missing-data constraints, and lists possible uses.

C.1 Motivation

  • Tenrec combines overlapping users and items across four scenarios, multiple positive-feedback types, true negative feedback, and features beyond user and item IDs.The stated intended uses include cross-domain recommendation, transfer learning, multi-task learning, click-through-rate prediction, and context- or content-based recommendation.Source: §C.1 · p. 16

C.2 Composition

  • The four datasets contain 493,458,970 QK video, 11,722,249 QB video, 46,111,728 QK article, and 348,736 QB article user-item interactions.The article records include fields such as interaction counts, reading percentage, categories, item scores, and reading time.Source: How many instances are there in total… · pp. 16–17

How many instances are there in total (of each type, if appropriate)?

  • Tenrec is a sample drawn randomly from Tencent feeds platforms, requiring users to have at least five video clicks; representativeness was not tested.A small percentage of instances lack video category, user age, or user gender, information described as missing in the real system.Source: How many instances are there in total… · pp. 16–17

C.3 Collection Process

  • User behavior logs came from Tencent feeds platforms between September 17 and December 07, 2021; the collection mechanisms were unknown to the datasheet authors.Users were randomly drawn subject to the five-video-click requirement.Source: §C.3 · p. 18

C.4 Preprocessing/cleaning/labeling

  • The dataset anonymizes user and item IDs and converts user profiles into discrete or binary values.Source: §C.4 · p. 18

C.5 Uses

  • The datasheet reports no prior dataset uses and points to a repository listing papers or systems that use it.Source: §C.5 · pp. 18–19

What (other) tasks could the dataset be used for?

The dataset is described as suitable for a range of recommendation tasks and is publicly available online.

What (other) tasks could the dataset be used for?

C.6 Distribution

  • The dataset is publicly available on the internet; its distribution details are given in the paper.Source: §C.6 · p. 19

When will the dataset be distributed?

The datasheet gives a planned distribution date and specifies the dataset’s license and citation request.

When will the dataset be distributed?

Who will be supporting/hosting/maintaining the dataset?

The datasheet names two dataset maintainers.

Who will be supporting/hosting/maintaining the dataset?

How can the owner/curator/manager of the dataset be contacted (e.g., email address)?

Dataset updates are not described as following a set schedule, and older versions are not maintained. Update details are posted on GitHub, while proposed fixes or extensions should be directed to the original authors.

Related papers

Loading 2210.10629v3…