Source-linked AI summary
KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems
Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao, Tat-Seng Chua
TL;DR
Recommender evaluation is difficult because online testing is costly and existing interaction datasets are sparse and biased. The paper introduces KuaiRec, a nearly fully observed real-world dataset, and studies how density and exposure bias affect conversational recommendation evaluation. It finds that method rankings vary with these factors, while matrix completion alleviates the problem only in certain cases.
Problem
Online testing is costly, while sparse and biased interaction data leave the reliability of offline conversational-recommendation evaluation uncertain.
Method
The paper introduces KuaiRec, a nearly fully observed dataset, and uses it to synthesize partially exposed data for studying density and exposure-bias effects.
Results
Performance rankings of conversational recommenders vary with data density and exposure bias, while matrix completion alleviates the problem only in certain cases.
Takeaways & Limitations
The findings demonstrate the significance of fully observed data for evaluating recommendation methods and studying trustworthy user simulation.
Takeaways & Limitations
The MTG setting lacks a CRS designed specifically for it, although MTG is more realistic in real life.
Abstract
from arXiv · showhide
The progress of recommender systems is hampered mainly by evaluation as it requires real-time interactions between humans and systems, which is too laborious and expensive. This issue is usually approached by utilizing the interaction history to conduct offline evaluation. However, existing datasets of user-item interactions are partially observed, leaving it unclear how and to what extent the missing interactions will influence the evaluation. To answer this question, we collect a fully-observed dataset from Kuaishou's online environment, where almost all 1,411 users have been exposed to all 3,327 items. To the best of our knowledge, this is the first real-world fully-observed data with millions of user-item interactions. With this unique dataset, we conduct a preliminary analysis of how the two factors - data density and exposure bias - affect the evaluation results of multi-round conversational recommendation. Our main discoveries are that the performance ranking of different methods varies with the two factors, and this effect can only be alleviated in certain cases by estimating missing interactions for user simulation. This demonstrates the necessity of the fully-observed dataset. We release the dataset and the pipeline implementation for evaluation at https://kuairec.com
1 INTRODUCTION
Offline evaluation avoids the cost and risk of online A/B tests, but sparse, biased interaction data makes its results difficult to trust. KuaiRec addresses this gap with a nearly fully observed dataset and uses it to study evaluation effects.
- Online A/B testing is time- and money-consuming and entails failure risk, motivating offline evaluation on observed interaction data.
- Sparse interaction matrices contain unknown values and exposure biases, making offline evaluation results difficult to persuade.The biases include popularity bias and positivity or selection bias.
- Yahoo! and Coat provide missing-complete-at-random data, but their remaining sparsity can produce high-variance evaluation results.
- KuaiRec combines a fully observed small matrix for faithful evaluation with a larger big matrix containing additional user-item interactions for training.
- KuaiRec is presented as the first real-world fully observed recommendation dataset, with millions of interactions and richer scale than existing MCAR datasets.
- The dataset supports analysis of how data density and exposure bias affect recommendation evaluation, including whether matrix completion mitigates the effects.
2 RELATED WORK
Prior offline recommendation evaluation relies on sparse data and strong assumptions, while conversational systems require user simulation before online deployment. KuaiRec provides a fully observed testbed for examining these evaluation challenges.
- Online evaluation is often impractical for academic researchers because A/B tests consume substantial time and money.
- Offline metrics such as Precision, Recall, NDCG, and MAP rely on assumptions that conflict with recommendation as a sequential decision problem.
- Off-policy evaluation estimates target-policy performance from another policy’s data, whereas user simulation fills missing values before evaluation.
- Existing unbiased datasets remain sparse; Yahoo! contains only 54,000 random interactions among 5,400 × 1,000 user-item pairs, a density of 1%.
- Conversational recommender systems ask attribute questions to narrow candidates, while multi-turn conversational recommendation focuses on interaction logic without raw-language processing.
- User simulators for conversational recommender evaluation are typically learned from sparse, biased data, motivating KuaiRec as a testbed for trustworthy evaluation.
3 DATA COLLECTION
KuaiRec collects a nearly fully observed small matrix from Kuaishou and a larger big matrix for training, then validates the small matrix’s representativeness.
- The small matrix: 1,411 users and 3,327 videos form the small matrix, whose density is 99.6%.The remaining missing values arise because some users explicitly blocked videos from certain authors.
- Dataset construction: The dataset collection process is described as producing a fully observed small matrix and a larger big matrix for training.The small matrix supports evaluation, while the big matrix contains additional users, videos, and interactions.
- The big matrix: The big matrix contains 7,176 users and 10,728 videos, including all users and videos in the small matrix.Interactions from the small matrix are excluded from the big matrix to separate training and evaluation data.
- Feedback representation: User-video feedback is binarized as positive when cumulative watch time exceeds twice the video duration.This criterion indicates that a user watched the video completely at least twice; raw likes were excluded because they were too sparse.
- Representativeness testing: Representativeness is tested with discrete two-sample KS tests over six selected user and video features.All reported p-values exceed the 5% rejection level, so the null hypothesis of equal feature distributions cannot be rejected.
4 ENVIRONMENT SETTING FOR CRS
The evaluation environment uses multi-round conversational recommendation and synthesizes partially observed data under different densities and exposure biases.
- Multi-round conversational recommendation: In MCR, each turn either recommends top-K items or asks about an item attribute, receiving binary user feedback.Negative feedback removes corresponding candidates, while positive attribute feedback filters out items lacking that attribute.
- User simulation: User simulators provide feedback from preferences estimated using observed data, but the effects of sparsity and bias remain unknown.The study therefore examines how partially observed data affects CRS evaluation.
- Partially observed data: The study varies data density and exposure strategy to examine their effects on CRS performance and model rankings.Three exposure strategies are used: uniformly random, positivity-oriented, and popularity-oriented exposure.
- Uniform exposure: Uniform random sampling creates nine missing-at-random datasets with densities from 10% through 90%.These datasets are used to explore how data density affects CRS evaluation without exposure biases.
- Popularity-oriented exposure: Popularity-oriented exposure assigns item probabilities using Zipf’s law with s=0.5 and N=3,327.This produces nine popularity-biased MNAR datasets at densities from 10% through 90%.
5 EXPERIMENTS
The experiments evaluate conversational recommender systems on the KuaiRec dataset and analyze their behavior under the study’s experimental conditions.
- The paper conducts intensive experiments on conversational recommender systems using the KuaiRec dataset.
- The experimental section focuses on evaluating conversational recommender systems with KuaiRec.
- KuaiRec serves as the dataset for the paper’s experiments on conversational recommender systems.
5.1 Experimental Setting
The experiments compare CRS evaluation on partially observed data before and after matrix completion, using multi-round conversations with simulated users and multiple ground-truth settings.
- Research questions: The experiments ask how biased or partially exposed data affects CRS evaluation and whether matrix completion improves it.
- Experimental configuration: Conversations last at most 15 rounds, recommend 10 items per recommendation turn, and compare results before and after matrix completion.
- User simulation: The study simulates users from partially exposed matrices created from the small matrix.
- Ground truth: MTG allows multiple ground-truth items per session, avoiding the STG treatment of other liked items as negatives.
- Metrics: CRS performance is measured by Average Turns and Success Rate SR@t, which captures conversations ending by round t.
- Baselines: Four representative CRS baselines share a factorization-machine recommendation engine and differ in their conversation strategies.
- Additional baselines: Additional baselines include Random, PopRec, and PosRec, which recommend random, popular, or broadly positive candidate items.
5.2 Exploring the Effects of Partially-Exposed Data in CRS Evaluation
Evaluation outcomes depend on both data density and exposure strategy, with the effects differing sharply between single-target and multi-target ground-truth settings. Uniform random exposure is relatively stable in STG, whereas MTG produces substantial variation even without exposure bias.
- Experimental setup: 10 repeated evaluations compare eight CRS methods across data densities and exposure strategies, first in STG and then under identical MTG configurations.All methods were trained on the big matrix before evaluation.
- STG setting: Under uniformly random exposure in STG, method rankings remain unchanged and performance is nearly stable across densities, except for Abs Greedy.The authors attribute this stability to independently and identically distributed positive samples and the law of large numbers.
- STG setting: Under positivity-oriented exposure in STG, PosRec and EAR deteriorate as density increases because additional lower-positivity items become harder for their positive-item-focused recommendations to predict.Other methods show relatively stable performance and consistent rankings in this setting.
- STG setting: Under popularity-oriented exposure in STG, PopRec has unstable performance and inconsistent rankings as increasing density exposes unpopular items that its popularity-focused strategy predicts poorly.Other methods remain stable and consistent, indicating lower sensitivity to popularity bias in STG.
- MTG setting: In MTG, both absolute performance and method rankings vary substantially across testing data even under uniformly random exposure.The setting is more complex because success requires finding at least one of multiple ground-truth items while estimating preferences from diverse feedback.
- MTG setting: Attribute questioning affects MTG evaluation more strongly than STG evaluation, with AttrAskRec improving at high density and CRM eventually outperforming Max Entropy on AT.CRM asks questions in 81.7% of turns versus 66.5% for Max Entropy; positive responses can narrow candidate items more quickly when density is high.
- MTG setting: Positivity and popularity biases alter MTG rankings more sharply, while uniform random sampling produces milder performance variation for Random and PosRec.Exposure bias can therefore aggravate disagreement among CRS rankings.
5.3 Effects of Estimating the Missing Values
The paper evaluates whether matrix completion can recover missing interactions and restore conversational recommender rankings from partially exposed data. It finds that completion helps in some settings but cannot reliably remove ranking inconsistencies or exposure-bias effects.
- Experimental Setup: Matrix completion estimates missing interactions for user simulation and evaluates conversational recommenders on the resulting completed matrix.The study selects ExpoMF-cov because it performs best in the authors’ experiments, while the conversational recommenders are pretrained on the big matrix.
- Matrix Completion Results: PMF performs worse than ExpoMF and ExpoMF-cov, especially under uniformly random and popularity-oriented exposure.The debiased methods model whether a zero represents an unobserved event or a negative sample.
- Matrix Completion Results: All three methods decline under positivity-oriented exposure as density increases because fewer positive samples remain available for prediction.The remaining unexposed positive interactions are concentrated among a few users and are difficult for the model to predict.
- Effects on Evaluation: Under random exposure, EAR’s inconsistent rankings decrease from 7 sampled datasets before completion to 5 after estimating missing values.Nearly half of the inconsistent-ranking counts decrease after completion, with the improvement most evident under random exposure.
- Effects on Evaluation: Matrix completion does not restore CRS rankings with respect to AT, although it can help when SR@15 performances are more distinguishable.The estimation error can blur small absolute-performance differences, whereas larger SR@15 differences are easier to recover.
- Effects of Biases: Exposure biases continue to affect CRS evaluation after missing-value estimation, and completion can sometimes introduce additional errors.Uniformly random samples generally have fewer inconsistent rankings than biased samples even after completion.
6 CONCLUSION
The paper presents KuaiRec, a fully observed recommender-systems dataset, and uses it to study how density and exposure bias affect evaluation. It concludes that the dataset can support trustworthy simulators and broader recommendation research, while correctly simulating fully observed data from partial observations remains open.
- Conclusion: KuaiRec is a fully observed dataset used to synthesize partially exposed data and study how data density and exposure bias affect recommendation evaluation.The paper reports extensive experiments on conversational recommender-system evaluation.
- Open Questions: The authors identify open questions about correctly simulating fully observed data from partially observed user-item interactions.Their matrix-completion experiments provide limited help, motivating further investigation with the fully observed data.
- Future Uses: KuaiRec is proposed as a testbed and benchmark for user simulators, debiasing, interactive recommendation, and faithful evaluation.The authors also encourage collecting fully observed datasets with richer domains and demographics.