Source-linked AI summary
Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System
Tianxin Wei, Fuli Feng, Jiawei Chen, Ziwei Wu, Jinfeng Yi, Xiangnan He
TL;DR
Popularity bias in recommender systems stems from interaction data and conventional training that favor popular items, while existing reweighting approaches are difficult to tune. MACR uses a causal graph, multi-task learning, and counterfactual inference to remove item popularity’s direct effect from ranking. Across five real-world datasets, it substantially improves MF and LightGCN, while its current formulation omits other causes because relevant side information or exposure mechanisms are unavailable.
Problem
Long-tailed interaction data and conventional behavior-reconstruction training bias recommenders toward popular items, while IPW methods are highly sensitive to weighting strategy.
Method
MACR formulates a causal graph, learns user and item effects through multi-task training, and removes item popularity’s direct ranking effect by counterfactual inference.
Results
Across five datasets, MACR improves MF by 153.13% in HR@20 on average and improves LightGCN by 241.98%.
Takeaways & Limitations
The model-agnostic framework is demonstrated on MF and LightGCN and is reported effective for reducing popularity bias.
Takeaways & Limitations
The current method focuses on interaction-frequency bias and omits other causes because side information and exposure mechanisms are unavailable.
Abstract
from arXiv · showhide
The general aim of the recommender system is to provide personalized suggestions to users, which is opposed to suggesting popular items. However, the normal training paradigm, i.e., fitting a recommender model to recover the user behavior data with pointwise or pairwise loss, makes the model biased towards popular items. This results in the terrible Matthew effect, making popular items be more frequently recommended and become even more popular. Existing work addresses this issue with Inverse Propensity Weighting (IPW), which decreases the impact of popular items on the training and increases the impact of long-tail items. Although theoretically sound, IPW methods are highly sensitive to the weighting strategy, which is notoriously difficult to tune. In this work, we explore the popularity bias issue from a novel and fundamental perspective -- cause-effect. We identify that popularity bias lies in the direct effect from the item node to the ranking score, such that an item's intrinsic property is the cause of mistakenly assigning it a higher ranking score. To eliminate popularity bias, it is essential to answer the counterfactual question that what the ranking score would be if the model only uses item property. To this end, we formulate a causal graph to describe the important cause-effect relations in the recommendation process. During training, we perform multi-task learning to achieve the contribution of each cause; during testing, we perform counterfactual inference to remove the effect of item popularity. Remarkably, our solution amends the learning process of recommendation which is agnostic to a wide range of models -- it can be easily implemented in existing methods. We demonstrate it on Matrix Factorization (MF) and LightGCN [20]. Experiments on five real-world datasets demonstrate the effectiveness of our method.
1 INTRODUCTION
Popularity bias arises because interaction data are long-tailed and conventional training favors popular items, while existing reweighting methods lack a systematic causal account. MACR models these causes explicitly, learns their contributions through auxiliary tasks, and removes item popularity’s direct effect by counterfactual inference.
- Motivation: Long-tailed interaction frequencies make conventional reconstruction training biased toward recommending popular items.The interaction distribution is shaped by exposure, word-of-mouth, campaigns, and item quality.
- Motivation: Popularity-biased recommendations can prioritize item popularity over user-item matching, reducing preference understanding and recommendation diversity.The training objective favors frequent popular-item recommendations because they can achieve lower loss.
- Motivation: Existing long-tail approaches downweight popular-item influence but lack fine-grained, interaction-specific modeling of how popularity produces bias.Examples include loss reweighting, balanced training data, and disentangling user and item embeddings.
- Causal formulation: MACR formulates a causal graph containing user-item matching, item popularity, and user conformity as causes of interaction probability.The graph adds direct user-to-score and item-to-score effects to represent user- and item-specific influences.
- Method: Counterfactual debiasing subtracts the score from a world without user-item matching, thereby removing the direct item-to-score effect during inference.The counterfactual world retains effects attributed to item popularity and user conformity.
- Method: The model jointly trains the main recommendation task with auxiliary tasks for user-to-score and item-to-score effects, and applies the framework to MF and LightGCN.The framework is designed to be model-agnostic and is evaluated on five real-world datasets.
2 PROBLEM DEFINITION
The problem is to learn a scoring function for user-item preference from interaction data while accounting for popularity bias in both training and testing. Debiased evaluation samples items uniformly to focus more on user preference and assess bias handling.
- Problem setup: Users and items are represented as sets U and I, and interactions form a matrix Y ∈ R^n×m.Here n and m denote the numbers of users and items, respectively.
- Problem setup: Recommender training learns a scoring function f(u,i|θ) that predicts a user’s preference for item i.The model is evaluated on holdout interactions during testing.
- Evaluation: Traditional holdout evaluation may not reflect true preference because popularity bias affects both training and testing data.The stated goal is to evaluate preference prediction despite this bias.
- Causal notation: In causal graphs, I, Y, and K can denote a cause, an effect, and a mediator, while gray nodes indicate reference-status variables.An I→K→Y path represents an indirect effect through mediator K.
- Evaluation: Debiased evaluation samples items to form a uniform distribution over items, enabling assessment of popularity-bias handling.This protocol follows prior work cited in the paper.
3 METHODOLOGY
MACR models recommendation causally, separating user, item, and user-item matching effects. It trains these components jointly and uses counterfactual inference to remove the direct popularity effect from ranking scores.
- Causal View: Causal graphs represent user, item, matching, and ranking-score variables, distinguishing direct effects from indirect effects through matching.The path I→K→Y is indirect, while I→Y is direct.
- Causal View: Popularity bias arises because item popularity directly raises interaction likelihood and ranking scores, causing recommendations to reflect popularity rather than user-item matching.Blocking the direct I→Y path is intended to eliminate this bias.
- MACR Framework: MACR adds user and item modules to existing recommenders, producing separate scores for user propensity, item popularity, and user-item matching.The framework is designed for models with the U&I→K→Y structure and uses multi-layer perceptrons for the added modules.
- MACR Framework: The three branch scores are aggregated into a final prediction, with sigmoid-scaled user and item scores modulating reliance on the matching score.For inactive users and unpopular items, training can enlarge the matching contribution to recover historical interactions.
- MACR Framework: Multi-task training adds supervision for the user and item modules alongside the overall recommendation objective, using trade-off hyper-parameters α and β.The additional losses are recommendation losses applied to the separate module scores.
- Counterfactual Inference: Counterfactual inference estimates the direct item-to-score effect by blocking matching with a reference item state, then removes that effect from recommendation scores.The counterfactual world assigns scores without user-item matching; c represents the reference status of the matching score.
4 EXPERIMENTS
The experiments evaluate MACR against debiasing baselines, hyper-parameter settings, framework components, and popularity-bias effects across five real-world datasets. MACR consistently improves MF and LightGCN while reducing recommendations of popular items and increasing recall for those items when appropriate.
- Experimental Setup: The evaluation uses five real-world datasets and a debiased testing protocol because conventional holdout interactions retain long-tail popularity bias.The datasets are ML10M, Adressa, Globo, Gowalla, and Yelp; evaluation follows prior work in simulating debiased recommendation.
- Experimental Setup: MACR is compared with MF, LightGCN, ExpoMF, CausE, BS, Reg, and IPW variants using HR@20, Recall@20, and NDCG@20.Table 2 evaluates methods at K=20, with Rec denoting Recall and statistically significant MACR improvements reported at p-value << 0.05.
- Overall Recommendation Performance: MACR consistently outperforms all compared methods, improving MACR_MF over MF by 153.13% and MACR_LightGCN over LightGCN by 241.98% in average HR@20.The averages are computed across the five datasets.
- Hyper-parameter Sensitivity: As c increases from 0 to 29, MACR_LightGCN improves on Adressa HR@20 before further increases become counterproductive; MACR_LightGCN is more sensitive than MACR_MF.The parameter controls how strongly intermediate matching preference is blocked during prediction.
- Component Ablation: Removing either branch reduces performance, with removing the item branch or its training loss harming results more than removing the corresponding user component.The ablation results indicate that item popularity bias has more influence than user conformity, while prediction fusion remains important.
- Debias Capability: MACR reduces recommendations of popular items and recommends more less-popular items, while the most popular group receives the greatest recall increase.The results suggest popular items are recommended more selectively, whereas unpopular items show smaller gains because they are sparsely represented in training data.
5 RELATED WORK
Related work addresses popularity bias through reweighting, debiased data, probabilistic exposure modeling, and ranking adjustment, while MACR uses causal reasoning with multi-task training and counterfactual inference.
- Popularity Bias in Recommendation: IPW methods reweight interactions in the training loss, often assigning lower weights to popular items.These methods include weighting interactions by the inverse of item popularity.
- Popularity Bias in Recommendation: Existing popularity-bias methods often do not model how popularity affects each specific interaction.The paper identifies this as a limitation of both IPW-based approaches and related debiasing strategies.
- Popularity Bias in Recommendation: Ranking-adjustment methods improve the rank or coverage of long-tail items but can trade recommendation accuracy for unpopular-item coverage.The paper characterizes brute-force promotion of long-tail items as typically causing an accuracy drop.
- Causal Inference in Recommendation: MACR captures popularity bias through multi-task training and removes it through counterfactual inference during prediction.This cause-effect approach differs from methods that directly reweight data or adjust rankings.
- Causal Inference in Recommendation: Causal inference analyzes relationships between causes and effects and has been applied across several machine-learning tasks, including recommendation.The paper positions causal reasoning as a foundation for its popularity-debiasing framework.
6 CONCLUSION AND FUTURE WORK
The paper presents MACR as a cause-effect framework for popularity debiasing and reports effectiveness across five real-world recommendation datasets. Future work expands causal reasoning to richer modules, side information, additional biases, and multiple simultaneous biases.
- Conclusion: MACR uses multi-task training on a causal graph to assess how different causes contribute to ranking scores.The framework is described as model-agnostic and applies counterfactual inference to estimate and remove the direct effect of item properties.
- Conclusion: MACR removes the direct effect of item properties from ranking scores through counterfactual inference.The stated goal is to eliminate popularity bias in recommendation.
- Conclusion: MACR’s effectiveness is demonstrated through extensive experiments on five real-world recommendation datasets.
- Future Work: Future work will extend causal reasoning to other recommender applications and improve user-conformity and item-popularity modules.
- Future Work: Future directions include incorporating side information, addressing other biases, eliminating multiple biases simultaneously, and combining causation with relational domain knowledge.
A INFERENCE PROCEDURE
The inference procedure describes MACR and traditional recommendation as algorithmic alternatives, with MACR receiving backbone, item, and user modules plus a reference status to produce a predicted user-item score.
- Inference Procedure: Algorithm 1 describes both the MACR procedure and the traditional recommendation system.
- Inference Procedure: The procedure takes a backbone recommender, item module, user module, user, item, and reference status as inputs.
- Inference Procedure: The procedure outputs the predicted user-item score ˆy_ui.
- Inference Procedure: The algorithm labels its MACR branch as Model Agnostic Counterfactual Reasoning and separately identifies the traditional recommender branch.
B IMPLEMENTATION DETAILS
The implementation uses common TensorFlow training settings across models, with specified exceptions for ExpoMF and larger datasets, and includes sensitivity analyses for MACR_MF parameters α and β.
- Training Setup: MACR is implemented in TensorFlow with embedding size 64 and Xavier initialization for all models.
- Training Setup: All models use Adam, learning rate 0.001, mini-batch size 1024, and binarized cross-entropy loss, except ExpoMF’s probabilistic training.The mini-batch size increases to 8192 on ML10M and Globo.
- Training Setup: LightGCN uses two layers, training lasts 1000 epochs, and the default L2 regularization coefficient is 1e-5.
- Parameter Analysis: Table 4 evaluates the effect of α on MACR_MF.
- Parameter Analysis: Table 5 evaluates the effect of β on MACR_MF.
C.1 Metrics with different Ks
On Adressa, MACR improves MF and LightGCN across HR@K, NDCG@K, and Recall@K for K values from 1 to 20; the other four datasets show the same trend.
- MACR improves MF and LightGCN on HR@K, NDCG@K, and Recall@K across K ∈ {1, 5, 10, 15, 20}.The reported gains are large across the evaluated metrics.
- The results shown for Adressa are representative of the same improvement trend observed on the other four datasets.
- The evaluation compares MACR-enhanced versions of both MF and LightGCN against their corresponding base models.
C.2 Effect of hyper-parameters
The experiments vary α and β separately on Adressa to examine the effects of the item and user auxiliary losses in MACR_MF.
- α balances the recommendation-model loss against the item-model loss in the joint objective.The experiments vary α over {1e-5, 1e-4, 1e-3, 1e-2}.
- β balances the recommendation-model loss against the user loss.The experiments vary β over {1e-5, 1e-4, 1e-3, 1e-2}.
- The ablation-style experiments use MACR_MF on the Adressa dataset and vary one hyper-parameter at a time.