Source-linked AI summary
Beyond Parity: Fairness Objectives for Collaborative Filtering
Sirui Yao, Bert Huang
TL;DR
Collaborative-filtering recommenders can inherit historical and sampling biases, producing unfair predictions even when observed ratings accurately reflect user preferences. The paper introduces four recommendation-specific unfairness metrics and fairness-penalized learning objectives. Experiments show that these unfairness forms can be minimized without significant reconstruction-error increases, although generalization depends on data density.
Problem
Historical population imbalance and observation bias can produce unfair recommendations in collaborative filtering, including when observed ratings accurately reflect users’ preferences.
Method
The paper proposes four unfairness metrics and augments matrix-factorization objectives with their penalties as regularizers.
Results
Minimizing the proposed unfairness penalties is possible on synthetic and real data without significant increases in reconstruction error.
Takeaways & Limitations
Practitioners should select the fairness objective corresponding to the specific unfairness form most important in their application because no single objective is best for all metrics.
Takeaways & Limitations
When ratings are especially sparse, empirical fairness does not always generalize well to held-out predictions.
Abstract
from arXiv · showhide
We study fairness in collaborative-filtering recommender systems, which are sensitive to discrimination that exists in historical data. Biased data can lead collaborative-filtering methods to make unfair predictions for users from minority groups. We identify the insufficiency of existing fairness metrics and propose four new metrics that address different forms of unfairness. These fairness metrics can be optimized by adding fairness terms to the learning objective. Experiments on synthetic and real data show that our new metrics can better measure fairness than the baseline, and that the fairness objectives effectively help reduce unfairness.
1 Introduction
The paper examines unfairness in collaborative-filtering recommendations caused by biased and imbalanced historical data. It proposes new fairness measures and optimization objectives to reduce these forms of unfairness.
- The paper formalizes recommendation unfairness, identifies demographic parity as insufficient, and proposes four metrics for different unfairness forms.
- The study compares the proposed measures with non-parity on biased synthetic data and evaluates fairness objectives on synthetic and real data.
- Collaborative filtering predicts user preferences from other users’ ratings or behavior, using aggregated opinions to recommend items.
- Non-random missing and observed ratings, influenced by social bias, can create unfairness in collaborative-filtering predictions.
- The paper focuses initially on basic matrix factorization, leaving the fairness effects of side-information models for future research.
2 Related Work
Prior fairness work includes demographic parity and equal opportunity, but demographic parity can be inappropriate when recommendations reflect genuine preference differences. The paper builds on fairness measures that respect group differences while evaluating recommendation-specific unfairness.
- Algorithmic fairness research has addressed supervised classification, while removing sensitive features is often insufficient because correlated attributes can preserve disparities.
- Demographic parity requires identical proportions of protected-group members receiving positive or negative classifications.
- Demographic parity is appropriate only when preferences are unrelated to sensitive features, an assumption that may fail in recommendation.
- Equal opportunity measures unfairness through equal true positive and true negative rates across protected groups.
- The equal-opportunity constraint respects group differences and penalizes models that perform well only for majority groups.
3 Fairness Objectives for Collaborative Filtering
This section explains how matrix-factorization collaborative filtering can produce unfair recommendations from population imbalance and observation bias, then introduces four metrics targeting distinct discrepancies between disadvantaged and advantaged users. It also describes the metrics’ prediction-error interpretations and their practical consequences.
- 3.1 Matrix Factorization for Recommendation: Matrix factorization represents ratings using user and item vectors with user and item bias terms, learned by minimizing regularized reconstruction error.The experiments use Adam, a gradient-based optimizer with adaptive learning rates and momentum.
- 3.2 Unfair Recommendations from Underrepresentation: Population imbalance and observation bias can each produce unfair recommendations, while their combination leads to worse unfairness.Population imbalance concerns group frequencies; observation bias concerns differing tendencies to rate item types, often within recommendation feedback loops.
- 3.3 Fairness Metrics: The paper introduces four metrics that measure discrepancies between disadvantaged and advantaged users’ prediction behavior, each capturing a different form of unfairness.The metrics are value, absolute, underestimation, and overestimation unfairness; a non-parity measure from prior work is also discussed.
- 3.3 Fairness Metrics: Value unfairness measures differences in signed estimation error, becoming large when one user class is consistently overestimated and the other underestimated.It becomes small when errors balance in direction or when both classes have errors with the same direction and magnitude.
- 3.3 Fairness Metrics: Absolute unfairness measures differences in absolute estimation error, regardless of whether predictions overestimate or underestimate true preferences.It is high when one user type has substantially larger reconstruction error than another, even if error directions differ.
- 3.3 Fairness Metrics: Underestimation and overestimation unfairness measure group differences in underpredicting and overpredicting true ratings, respectively.Underestimation matters when missed recommendations are more critical; overestimation matters when excessive recommendations burden users’ time.
4 Experiments
Experiments on synthetic and movie-rating data test whether underrepresentation produces unfair recommendations and whether fairness-augmented objectives reduce distinct unfairness forms. Results show that population and observation bias increase unfairness, while optimizing individual metrics generally improves them without significantly worsening reconstruction error.
- Synthetic Data: Synthetic experiments use block-model ratings with four user groups, three item groups, and controlled population and observation imbalance.The study generates 400 users and 300 items, then evaluates predictions on unseen matrix entries.
- Unfairness from different types of underrepresentation: For most metrics, fairness worsens from uniform data to biased observations, biased populations, and finally combined biases.Squared rating error follows the same ordering, whereas non-parity is amplified mainly by biased observations and is less informative about predictor unfairness in that setting.
- Optimization of unfairness metrics: Fairness objectives successfully minimize their targeted unfairness penalties on held-out user-item pairs without significantly increasing reconstruction error.The synthetic evaluation measures squared reconstruction error and six unfairness metrics across fairness objectives.
- Optimization of unfairness metrics: Optimizing one unfairness metric usually reduces other forms, but absolute unfairness can increase underestimation and parity optimization can increase most other metrics.Value unfairness is especially effective at reducing underestimation and overestimation, so practitioners must choose fairness priorities for their applications.
- Real Data: The real-data study filters MovieLens ratings to 2,953 users and 1,006 movies across action, crime, musical, romance, and sci-fi genres with gender-related differences.Five random train-test trials evaluate each fairness objective on the selected ratings.
- Real Data: On real movie-rating data, each objective achieves the best performance on its corresponding unfairness metric without significantly changing reconstruction error.Value unfairness again most reduces under- and overestimation, while non-parity increases or leaves nearly all other metrics unchanged.
5 Conclusion
The paper shows that collaborative filtering can exhibit multiple forms of unfairness and that fairness penalties can reduce them without significantly increasing reconstruction error. It also identifies practical scope boundaries involving objective selection, item-group fairness, data sparsity, and rating assumptions.
- 5 Conclusion: Fairness penalties added to matrix-factorization objectives enable learning algorithms to minimize each identified form of unfairness.This result is demonstrated for the paper’s collaborative-filtering setting.
- 5 Conclusion: Minimizing the identified unfairness forms is possible on synthetic and real data with no significant increase in reconstruction error.
- 5 Conclusion: No single objective performs best across all unfairness metrics, so practitioners must select the fairness form most important to their application.
- 5 Conclusion: The study addresses fairness among user groups but not fair treatment of different item groups.The paper notes that item groups may also experience discrimination or bias.
- 5 Conclusion: Generalization from empirical unfairness to held-out predictions depends on data density and may fail when ratings are especially sparse.
- 5 Conclusion: The fairness metrics assume ratings reflect users’ true preferences, an assumption that environmental factors can violate in real data.Addressing such bias may require additional information or external interventions beyond rating data.