Source-linked AI summary
KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos
Chongming Gao, Shijun Li, Yuan Zhang, Jiawei Chen, Biao Li, Wenqiang Lei, Peng Jiang, Xiangnan He
TL;DR
Exposure-biased logs and missing interactions complicate unbiased offline evaluation of recommender systems. KuaiRand collects randomly exposed videos within Kuaishou feeds, recording rich feedback, histories, and side information. The dataset supports large-scale debiasing and related sequential recommendation research.
Problem
Exposure-biased logs and missing interactions limit unbiased offline evaluation of recommender systems.
Method
KuaiRand inserts randomly chosen videos into normal Kuaishou feeds and records 12 feedback signals, user histories, and rich user/item side information.
Results
KuaiRand provides millions of intervened interactions and enables large-scale sequential recommendation debiasing and unbiased offline evaluation.
Takeaways & Limitations
The dataset supports research on debiasing, interactive recommendation, long sequential behavior modeling, and multi-task learning.
Takeaways & Limitations
Exact values are hidden for commercial security reasons to avoid leaking key parameters of the online recommendation strategy.
Abstract
from arXiv · showhide
Recommender systems deployed in real-world applications can have inherent exposure bias, which leads to the biased logged data plaguing the researchers. A fundamental way to address this thorny problem is to collect users' interactions on randomly expose items, i.e., the missing-at-random data. A few works have asked certain users to rate or select randomly recommended items, e.g., Yahoo!, Coat, and OpenBandit. However, these datasets are either too small in size or lack key information, such as unique user ID or the features of users/items. In this work, we present KuaiRand, an unbiased sequential recommendation dataset containing millions of intervened interactions on randomly exposed videos, collected from the video-sharing mobile App, Kuaishou. Different from existing datasets, KuaiRand records 12 kinds of user feedback signals (e.g., click, like, and view time) on randomly exposed videos inserted in the recommendation feeds in two weeks. To facilitate model learning, we further collect rich features of users and items as well as users' behavior history. By releasing this dataset, we enable the research of advanced debiasing large-scale recommendation scenarios for the first time. Also, with its distinctive features, KuaiRand can support various other research directions such as interactive recommendation, long sequential behavior modeling, and multi-task learning. The dataset and its news will be available at https://kuairand.com.
1 INTRODUCTION
Offline recommendation research is constrained by exposure-biased interaction logs and missing user–item interactions. KuaiRand addresses this gap with randomly exposed videos, rich feedback, side information, and sequential histories.
- Offline recommendation data is affected by exposure bias, producing inconsistent online and offline model results.
- Missing interactions make it difficult to learn user preferences for items that were not previously exposed.
- Randomly sampled item feedback provides missing-at-random data that supports unbiased offline evaluation.
- Existing randomly exposed-item datasets are limited in scale or omit key information such as user/item features and timestamps.
- KuaiRand inserts randomly chosen videos into normal feeds and records 12 feedback signals, user histories, and rich user/item side information.
- KuaiRand’s unbiased sequential data supports large-scale debiasing, unbiased offline evaluation, interactive recommendation, long-sequence modeling, and multi-task learning.
2 RELATED WORK
Offline evaluation avoids costly online A/B testing but remains limited by missing data, high variance, or simulation error. KuaiRand compares favorably with prior randomly exposed-item datasets through richer and more comprehensive sequential information.
- Online A/B testing is effective but often too time- and money-consuming for academic recommendation evaluation.
- Offline evaluation includes traditional test-set indicators, off-policy evaluation, counterfactual reasoning, and user simulation.
- Traditional indicators cannot evaluate performance on missing user–item pairs, while off-policy evaluation has high variance and simulation introduces additional error.
- Collecting feedback on randomly exposed items is presented as a fundamental way to obtain unbiased offline-evaluation data.
- Prior datasets such as Yahoo!R3, Yahoo!R6, Coat, and Open Bandit differ in scale, timestamps, user IDs, features, or policy coverage.
- KuaiRand provides explicit IDs, timestamps, rich user/item features, 15 policies, 12 feedback signals, and thousands of historical interactions per user on average.
3 DATA DESCRIPTION
KuaiRand is collected from Kuaishou’s short-video recommendation scenarios by inserting uniformly sampled random videos into recommendation feeds. It combines intervened interactions with extensive normal histories and side information.
- 3.1 Characteristics of Kuaishou App: Kuaishou has over 300 million daily-active users, and short-video use generates very long daily interaction sequences.
- 3.1 Characteristics of Kuaishou App: Kuaishou contains 15 recommendation scenarios, each with its own policy, rules, and objectives.
- 3.1 Characteristics of Kuaishou App: Different interfaces make click, view time, like, author-profile entry, and author-follow signals informative indicators of user behavior or satisfaction.
- 3.2 Data Collection: The collection samples videos, filters spam, randomly selects users, and inserts uniformly sampled random items into recommendation lists with fixed probability.
- 3.2 Data Collection: 1,186,059 random interventions were recorded for 27,285 retained users, with all 7,583 items inserted at least once.
- 3.2 Data Collection: The one-month dataset includes 322,278,385 normal recommendations and historical interactions plus user and item side information.
- 3.3 Statistics and Usage: Most KuaiRand-27K users viewed thousands of recommended videos in one month and more than 10 randomly exposed videos.
- 3.3 Statistics and Usage: Exact values are withheld for commercial security reasons to avoid revealing key parameters of the online recommendation strategy.
4 POTENTIAL RESEARCH DIRECTIONS
KuaiRand supports debiasing and several sequential recommendation research directions by providing randomized exposure data with rich interaction and identity information. Its logs enable offline policy evaluation and other large-scale recommendation tasks.
- Debiasing in Recommendation: KuaiRand enables large-scale debiasing by inserting random items into normal feeds with rich side information and explicit user/item IDs.
- Debiasing in Recommendation: The dataset’s distinctive features facilitate unbiased offline evaluation and debiasing in large-scale recommendation scenarios.
- Other Research Directions: KuaiRand can support interactive recommendation, long sequential behavior modeling, and multi-task learning.
- Off-policy Evaluation (OPE): Its unbiased sequential logs support evaluating and improving recommendation policies without relying solely on online A/B tests.The dataset records logs under 15 policies, satisfying the sequential-log requirement for off-policy evaluation.
video_id user_id
KuaiRand distinguishes user and video identity spaces across three dataset versions and relates them to suggested research tasks. The candidate pool and viewed-video subsets define important data relationships.
- video_id user_id: The candidate pool contains videos that users may view, while normally viewed videos include recommendation-feed interactions.
- video_id user_id: The 1K-user subset retains the videos viewed by those selected users.
- video_id user_id: Figure 4 compares user/video ID spaces across three KuaiRand versions and maps them to suggested tasks.
A VERSIONS AND SUGGESTIONS
KuaiRand is released in three versions tailored to different research and resource requirements. They trade dataset scale and sequential completeness for easier computation or narrower task support.
- KuaiRand-27K: KuaiRand-27K contains 23GB of logs, 23GB of features, over 27K users, and 32 million videos for rigorous sequential research.It is suggested for off-policy evaluation, reinforcement learning, and long sequential recommendation.
- KuaiRand-1K: KuaiRand-1K contains 829MB of logs and 3.5GB of features for researchers unable to process the complete dataset.It uniformly selects 1,000 users from KuaiRand-27K and removes irrelevant users and videos.
- KuaiRand-Pure: KuaiRand-Pure contains 184MB of logs and 10MB of features, retaining only logs for 7,582 candidate-pool videos.It is intended for tasks that do not require rigorous sequential information, such as collaborative filtering debiasing, domain adaptation, and multi-task modeling.