Collaborative Filtering and the Missing at Random Assumption

Benjamin Marlin; Richard S. Zemel; Sam Roweis; Malcolm Slaney

Source-linked AI summary

Collaborative Filtering and the Missing at Random Assumption

Benjamin Marlin, Richard S. Zemel, Sam Roweis, Malcolm Slaney

arXiv:1206.5267v1Jun 20, 2012cs.LGcs.IRstat.ML

TL;DR

Collaborative-filtering learning and evaluation rely on the MAR assumption, yet user-selected ratings may differ systematically from random ratings. The paper studies this with LaunchCast data and finds that explicitly modeling missingness improves prediction on random ratings.

Problem
The paper examines whether the MAR assumption supporting collaborative-filtering learning and testing is violated in real user rating data.
Method
The authors analyze LaunchCast survey responses and random-song ratings, compare them with user-selected ratings, and introduce an explicit missing-data model and evaluation protocol.
Results
MM/CPT-v achieved a best average test error of 0.7148 versus 1.2126 for MM/None, reducing test error by over 40%.
Takeaways & Limitations
The study provides evidence against MAR and shows that modeling the missing-data mechanism can significantly improve rating prediction on random ratings.
Takeaways & Limitations
Even with strong prior information, estimated missing-data parameters produced significantly lower-quality solutions than parameters estimated from held-out ratings.

Abstract

from arXiv · show

Rating prediction is an important application, and a popular research topic in collaborative filtering. However, both the validity of learning algorithms, and the validity of standard testing procedures rest on the assumption that missing ratings are missing at random (MAR). In this paper we present the results of a user study in which we collect a random sample of ratings from current users of an online radio service. An analysis of the rating data collected in the study shows that the sample of random ratings has markedly different properties than ratings of user-selected songs. When asked to report on their own rating behaviour, a large number of users indicate they believe their opinion of a song does affect whether they choose to rate that song, a violation of the MAR condition. Finally, we present experimental results showing that incorporating an explicit model of the missing data mechanism can lead to significant improvements in prediction performance on the random sample of ratings.

1 Introduction

Collaborative-filtering rating prediction and its evaluation assume missing ratings are missing at random, but user-selected ratings can be systematically biased. The paper studies this issue with real LaunchCast users and models the missing-data mechanism explicitly.

1 Introduction: MAR underlies both collaborative-filtering learning methods and standard testing procedures.Violations can bias learned parameters and make observed-data test error a poor estimate of error on the fully completed data.
1 Introduction: When rating visibility depends on the rating value, observed collaborative-filtering data can overrepresent higher ratings.Users may preferentially see and rate items they expect to like, creating systematic selection bias.
1 Introduction: Non-random missingness can bias item predictions even when nearest-neighbour identities remain accurate.Conditioning on neighbours who rated the item can distort the predicted rating, and analogous bias can affect model parameters.
1 Introduction: The paper presents a real-user LaunchCast study of how the MAR assumption affects collaborative filtering.It analyzes survey responses and randomly chosen-song ratings, then develops models and an evaluation protocol for non-random missing data.
1 Introduction: Explicitly modeling the missing-data mechanism can significantly reduce test error relative to treating ratings as missing at random.The paper evaluates training on user-selected items and testing on randomly selected items.

2 Missing Data Theory

The missing-data framework distinguishes how ratings are generated from how observations are selected. MAR permits response probabilities to depend on observed data but not missing values; violating it can bias inference.

2 Missing Data Theory: A collaborative-filtering dataset is represented as ratings X, response indicators R, and latent variables Z.x_im is user i’s rating for item m, while r_im indicates whether that rating is observed.
2 Missing Data Theory: The joint model separates the data model P(X, Z|θ) from the missing-data mechanism P(R|X, Z, µ).The data model generates complete cases, and the missing-data model selects which matrix elements are observed.
2 Missing Data Theory: Under MAR, response probabilities may depend on observed data but not on the missing values themselves.MCAR forbids dependence on data values, whereas MAR conditions response patterns on the observed portion of the data vector.
2 Missing Data Theory: NMAR occurs when the probability of observing a rating depends on that rating’s value.In collaborative filtering, this means whether an item is rated can depend on the user’s unobserved rating for it.
2 Missing Data Theory: When MAR holds, observed-data maximum-likelihood inference is unbiased because response probabilities are independent of missing data.When MAR fails, the likelihood cannot be simplified in the same way, and parameter estimation may be biased.

3 Yahoo! LaunchCast Rating Study

The LaunchCast study collected random survey ratings alongside users’ existing ratings to examine how rating selection affects observed data. Survey ratings differed sharply from ratings for user-selected songs, while overlapping ratings showed good agreement.

Study design: The study required a random test set and collected ratings from LaunchCast users through a survey.Users rated ten songs randomly selected from a fixed set of 1000 songs; the study included 35,786 contributors, with most analyses restricted to 5,400 participants having at least 10 base ratings.
User survey: 64.85% of users reported that their preferences affect whether they choose to rate a song, contradicting the missing at random assumption.Users generally rated songs they loved more often than songs they felt neutral about, and somewhat more often than songs they hated.
Study design: Survey ratings sampled randomly chosen songs, whereas base ratings came from songs participants had chosen to rate.The base set consisted of existing ratings extracted from the LaunchCast database.
Rating distributions: The survey set contained many fewer four- and five-star ratings than the base set, while EachMovie, MovieLens, and Netflix also had higher proportions of high ratings than the random sample.The difference is consistent with users influencing LaunchCast to play songs reflecting their preferences.
Rating distributions: The median per-song symmetrised Kullback−Leibler divergence between survey and base rating distributions was 0.8750 bits.Half of the songs exhibited a more extreme difference according to this measure; song 838 had an SKL value of 0.8749 bits.
Consistency check: Ratings for overlapping user-song pairs agreed quite well, with approximately 1700 observations in the intersection of the survey and base sets.The comparison used all 35,786 survey participants.
Interpretation: Differences between survey and base marginal distributions are not conclusive evidence of NMAR because MAR may hold for the true data model but fail for a simpler marginal model.Combined with the user survey, the authors regard the findings as compelling evidence against MAR.

4 Modeling Non-Random Missing Data

The paper extends a Bayesian multinomial mixture model to account for non-random missing ratings by combining it with a CPT-v missing-data model. The combined model supports learning and prediction while representing how user preferences affect rating decisions.

4. Modeling Non-Random Missing Data: The paper combines the multinomial mixture model with a Bayesian CPT-v missing-data model to capture a key non-random missingness property suggested by user survey results.It provides learning and prediction methods for the combined mixture/CPT-v model.
4.1 Multinomial Mixture Data Model: The multinomial mixture model treats users as belonging to preference-based clusters and generates each user’s ratings from the corresponding cluster distribution.User cluster membership is latent, while rating values are sampled independently across items conditional on that membership.
4.1 Multinomial Mixture Data Model: The Bayesian mixture model learns parameters from incomplete data by maximizing posterior log probability with a MAP expectation maximization algorithm.The EM procedure computes posterior distributions over each user’s latent mixture indicator during its expectation step.
4.2 The CPT-v Missing Data Model: The CPT-v model captures the possibility that a user’s preference for an item affects whether the user rates it.The extended model assumes item-rating choices are independent and models the probability of rating value v with Bernoulli parameter µv.
4.2 The CPT-v Missing Data Model: The CPT-v extension uses a Beta prior for the Bernoulli parameters µv and integrates missing-data effects into posterior optimization.Its factorized structure permits missing values to be summed out, leaving local factors for individual missing-data values.
4. Modeling Non-Random Missing Data: Predictions first infer the posterior distribution over a user’s mixture indicator, then compute the predictive distribution of rating values for the target item.The paper specifies separate inference equations for the MAR mixture model and the mixture/CPT-v model before applying the predictive calculation.

5 Experimental Method and Results

The experiments test whether explicitly modeling non-random missingness improves rating prediction on randomly sampled ratings. They compare mixture models with and without a CPT-v missing-data model under a protocol separating user-selected training ratings from random test ratings.

Experimental Protocol: The protocol trains on possibly non-random existing ratings and tests on 50,000 ratings randomly sampled from 10 songs per survey participant.The test set covers 5,000 survey participants, while the training set includes existing ratings for 10,000 users.
Compared Models: The baseline is a Bayesian multinomial mixture model under MAR, compared with the same mixture model combined with the CPT-v missing-data model.Both models use MAP-EM; MM/None ignores the missing-data mechanism, whereas MM/CPT-v models it explicitly.
Missing-Data Model: The study estimates CPT-v observation probabilities from held-out survey ratings and previously existing ratings, yielding µ̂ = [0.014, 0.011, 0.027, 0.063, 0.225].The survey ratings estimate βv, while existing LaunchCast ratings estimate δv under the mechanism being modeled.
Rating Prediction Results: 0.7148 versus 1.2126 was the best average test error for MM/CPT-v and MM/None, respectively, a reduction of over 40% with ten versus one mixture components.The reported standard error of the mean is below 0.01 for both models.
Rating Prediction Results: MM/None test error increased slightly as mixture components increased, despite its training fit improving with model complexity.The authors attribute this pattern to closer matching of the training-data distribution at the expense of test performance.
Rating Prediction Results: When both mixture and missing-data parameters were learned, experiments varied mixture components from 1 to 10 and prior strength from 200 to 100,000.Five repetitions were performed for each combination; the reported maximum standard error of the mean was below 0.001.

6 Discussion and Conclusions

The study provides evidence that real collaborative-filtering ratings violate the missing-at-random assumption, and that explicitly modeling missingness can improve prediction. It also shows that random ratings can estimate missing-data parameters that generalize, although parameter learning remains inferior to estimation from held-out ratings.

Users’ preferences affect which songs they choose to rate, providing evidence against the missing-at-random assumption in real collaborative-filtering data.
The CPT-v missing-data model produced a surprising boost in test performance when suitable missing-data parameters could be learned or estimated.
A relatively small number of randomly chosen song ratings can estimate missing-data parameters that generalize well to a larger user population.
Learning missing-data parameters still yields significantly lower-quality solutions than estimating them using held-out ratings, even with strong prior information.
MCMC inference could improve prediction and support more flexible hierarchical and non-parametric data and missing-data models.