Source-linked AI summary
Counterfactual Explainable Recommendation
Juntao Tan, Shuyuan Xu, Yingqiang Ge, Yunqi Li, Xu Chen, Yongfeng Zhang
TL;DR
Explainable recommendation needs explanations that help users understand recommendations and help designers debug complex models, while existing methods often rely on associative signals without intervention-based reasoning. CountER uses counterfactual changes to item aspects to produce simple, effective explanations, and experiments on five real-world datasets validate its effectiveness, with better FNS performance than baselines.
Problem
Existing explainable-recommendation methods mainly extract associative signals such as feature importance and attention weights, while explainability is needed for user understanding, trust, decision making, and model debugging.
Method
CountER applies counterfactual reasoning and constrained learning to find minimal changes in item aspects that reverse a recommendation, balancing explanation complexity and strength.
Results
Experiments on five real-world datasets validate CountER’s effectiveness, while its FNS score improves over the best baseline by 15.63% with masking and 38.49% without masking.
Takeaways & Limitations
Counterfactual explanations can support quantitative evaluation while helping users understand recommendations and system designers debug recommendation models.
Takeaways & Limitations
CountER’s relaxed optimization does not guarantee that the counterfactual item leaves the top-K list, so successful removal must be checked in post-processing.
Abstract
from arXiv · showhide
By providing explanations for users and system designers to facilitate better understanding and decision making, explainable recommendation has been an important research problem. In this paper, we propose Counterfactual Explainable Recommendation (CountER), which takes the insights of counterfactual reasoning from causal inference for explainable recommendation. CountER is able to formulate the complexity and the strength of explanations, and it adopts a counterfactual learning framework to seek simple (low complexity) and effective (high strength) explanations for the model decision. Technically, for each item recommended to each user, CountER formulates a joint optimization problem to generate minimal changes on the item aspects so as to create a counterfactual item, such that the recommendation decision on the counterfactual item is reversed. These altered aspects constitute the explanation of why the original item is recommended. The counterfactual explanation helps both the users for better understanding and the system designers for better model debugging. Another contribution of the work is the evaluation of explainable recommendation, which has been a challenging task. Fortunately, counterfactual explanations are very suitable for standard quantitative evaluation. To measure the explanation quality, we design two types of evaluation metrics, one from user's perspective (i.e. why the user likes the item), and the other from model's perspective (i.e. why the item is recommended by the model). We apply our counterfactual learning algorithm on a black-box recommender system and evaluate the generated explanations on five real-world datasets. Results show that our model generates more accurate and effective explanations than state-of-the-art explainable recommendation models.
1 INTRODUCTION
CountER addresses limitations of matching-based explanations by finding minimal aspect changes that reverse a black-box recommender’s decision. It also proposes user- and model-oriented quantitative evaluation methods and validates the approach on five real-world datasets.
- Motivation: Existing aspect-aware methods often target intrinsic models, use fixed aspect counts, and rely on associative matching signals rather than interventions.These limitations make explanations difficult to apply to black-box recommenders and may miss multi-aspect reasons for recommendations.
- Counterfactual intuition: Counterfactual reasoning identifies aspects whose slight deterioration would reverse a recommendation, rather than simply selecting the aspect with the highest matching score.In the example, battery is selected because changing its score from 3 to 2.1 reverses the decision, whereas screen has the highest matching score.
- Proposed method: CountER jointly seeks explanations with low complexity and high strength by optimizing minimal changes to item aspects that alter the recommendation decision.Complexity measures the required change, while strength measures its influence on the recommendation.
- Evaluation: The paper introduces user-oriented and model-oriented evaluation metrics for explainable recommendation.User-oriented evaluation uses review-mentioned aspects, while model-oriented evaluation uses Probability of Necessity, Probability of Sufficiency, and FNS.
- Evaluation: The approach is evaluated through extensive experiments on five real-world datasets to validate its effectiveness.The contribution list identifies five real-world datasets and frames the experiments as validation of the proposed method.
2 RELATED WORK
Related work primarily uses aspect matching or other explainable recommendation models, while CountER instead learns item-side counterfactual explanations directly from recommendation changes.
- Explainable recommendation: Explainable recommendation includes aspect-aware, deep, set-based, social, tree-enhanced, and other model families.Aspect-aware approaches commonly extract item features and user opinions to construct explanations.
- Matching-based explanations: Many existing approaches explain recommendations by selecting the aspect with the strongest match between user preference and item performance.This matching hypothesis produces associative explanations rather than explanations based on altered recommendation outcomes.
- Counterfactual reasoning: Counterfactual reasoning has been applied in explainable AI and recommendation to reason from interventions rather than associations.Prior applications span computer vision, natural language processing, social fairness, and recommendation.
- CountER’s distinction: CountER generates counterfactual explanations on item aspects, whereas prior counterfactual recommendation work generates them from user-side actions.The paper distinguishes these as different explanation types.
- CountER’s distinction: CountER uses an Occam’s-Razor-driven counterfactual learning objective to directly seek explanations with small complexity and large strength.The desired simplicity and effectiveness are encoded in the model objective.
3 PROBLEM FORMULATION
The paper formulates counterfactual explanations for black-box recommendation as aspect changes that remove an item from a user’s top-K list. It formalizes complexity and strength, then optimizes for simple, effective explanations.
- 3.1 Preliminaries and Notations: The problem formulation represents users, items, interactions, top-K recommendation lists, and user- and item-aspect information extracted from reviews.The interaction matrix records user-item interactions, while aspect matrices encode preferences and item quality.
- 3.1 Preliminaries and Notations: User-aspect and item-aspect values are constructed from aspect frequencies and sentiment, then rescaled to the system’s rating range.The rating scale is typically five stars, and sigmoid rescaling maps matrix elements into (1, N).
- 3.2 Counterfactual Explainable Recommendation: For a recommended item, the objective is to find a nonpositive change vector Δ on its aspect-quality vector that removes the item from the top-K list.The counterfactual statement is: if the item had been slightly worse on selected aspects, it would not be recommended.
- 3.3 Explanation Complexity and Strength: Explanation Complexity combines the number of altered aspects, represented by ∥Δ∥0, with the squared magnitude of their changes, represented by ∥Δ∥2.A hyper-parameter γ controls the trade-off between these two components.
- 3.3 Explanation Complexity and Strength: Explanation Strength measures how much the intervention changes the recommended item’s ranking score.It is defined using the decrease from the original score s_i,j to the intervened score s_i,j^Δ.
- 3.3 Explanation Complexity and Strength: The framework can adapt to alternative definitions of explanation complexity and strength depending on practical needs.The paper presents the stated definitions as nonexclusive choices rather than the only possible formulations.
- 3.3 Explanation Complexity and Strength: Complexity and strength are orthogonal: a complex explanation can be weak, and a simple explanation can be strong.Under Occam’s Razor, equally effective explanations should favor the simpler one.
4 COUNTERFACTUAL REASONING
CountER explains recommendations for a black-box model by finding small changes to item aspects that make a recommended item leave the top-K list. Its optimization balances explanation complexity against strength, with relaxation and post-processing used to make the problem tractable.
- 4.1 Black-box Recommendation Model: CountER applies counterfactual reasoning to generate explanations for items recommended by an existing black-box recommendation model.The framework searches for changes to item aspects rather than requiring an intrinsically explainable recommender.
- 4.2 Counterfactual Reasoning: The counterfactual explanation changes an item's aspects so its ranking falls below the K+1 item and it is removed from the top-K list.The altered aspect values form the explanation of the original recommendation.
- 4.2 Counterfactual Reasoning: The optimization minimizes explanation complexity while requiring sufficient explanation strength to remove the item from the recommendation list.Complexity reflects the magnitude and number of changed aspects, while strength reflects the effect on the recommendation decision.
- 4.3 Relaxed Optimization: ℓ1 relaxation replaces the nonconvex ℓ0 term to encourage sparse aspect changes, while a hinge-loss relaxation makes the ranking constraint optimizable.The relaxed objective improves optimization but does not guarantee successful removal from the top-K list.
- 4.3 Relaxed Optimization: Successful counterfactual explanations require post-processing because relaxed optimization can fail to make the changed item's score lower than the K+1 item's score.The method reports fidelity as the percentage of items for which explanation generation succeeds.
- 4.4 Discussions: Increasing λ prioritizes explanation strength over complexity, and higher-ranked items generally require more aspects or larger changes to be removed.The framework can also enforce single-aspect explanations using a trainable one-hot mask.
5 EVALUATING EXPLANATIONS
The paper evaluates explanations from both user and model perspectives using quantitative counterfactual metrics. User-oriented metrics compare explanations with positively mentioned review aspects, while model-oriented metrics test necessity and sufficiency.
- 5 Evaluating Explanations: The evaluation framework defines user-oriented and model-oriented metrics for quantitative offline assessment of explainable recommendations.This addresses the expense, limited scalability, and lack of standardization associated with human-subject evaluation.
- 5.1 User-oriented Evaluation: User-oriented evaluation treats positively mentioned aspects in user reviews as ground-truth reasons for purchasing an item.The generated explanation consists of the aspects corresponding to nonzero entries in the change vector Δ.
- 5.1 User-oriented Evaluation: Precision measures the share of generated aspects liked by the user, while recall measures the share of liked aspects included in the explanation.The evaluation also averages these scores and their harmonic-mean F1 across user-item pairs.
- 5.2 Model-oriented Evaluation: User-oriented evaluation measures consistency with user preferences but does not establish whether an explanation justifies the recommendation model's behavior.Model-oriented evaluation is therefore needed to assess the recommendation mechanism itself.
- 5.2 Model-oriented Evaluation: Probability of Necessity tests whether removing the explanation aspects causes the recommended item to leave the recommendation list.It creates a counterfactual item set by setting the related aspect values to zero and rerunning recommendation.
- 5.2 Model-oriented Evaluation: Probability of Sufficiency tests whether the item remains recommended when the explanation aspects are the only aspects retained.PN and PS are combined with their harmonic mean FNS for an overall model-oriented measure.
6 EXPERIMENTS
CountER is evaluated on five real-world datasets using user-oriented and model-oriented explanation metrics, alongside fidelity, complexity, strength, and ablation analyses. It generally produces stronger explanations than baselines while revealing trade-offs between explanation coverage, complexity, and evaluation performance.
- Experimental Setup: CountER is evaluated on Yelp2 and four Amazon datasets, comparing three aspect-aware models and a random explanation baseline.The Amazon datasets are Electronic, Cell Phones and Accessories, Kindle Store, and CDs and Vinyl; preprocessing removes users with few reviews.
- Explanation Fidelity: Multi-aspect explanations achieve 80%–100% fidelity in most cases, while single-aspect fidelity is lower; their average complexity is 2.79 aspects.Fidelity measures the percentage of recommended items for which the method successfully generates an explanation.
- User-oriented Evaluation: With masking for fair comparison, CountER outperforms all baselines on F1 across datasets, with higher precision in 90% and recall in 80% of cases.The reported Yelp improvement is especially large, and the authors associate this with Yelp’s greater density and review volume.
- Model-oriented Evaluation: CountER exceeds all baselines on FNS with or without masking, improving over the best baseline by 15.63% with masking and 38.49% without it.Masking limits CountER’s ability to explain the model’s behavior, but CountER remains strongest on FNS.
- Model-oriented Evaluation: Matching-based baselines often have higher PS but much lower PN, because aligned aspects can preserve recommendations while removing them may leave other competitive aspects.This pattern is presented as evidence that matching-based explanations may miss the reasons behind recommendations.
- Influence of Relaxation Weight λ: Increasing λ raises fidelity and explanation complexity, while worsening user-oriented performance; explanation strength and model-oriented performance remain largely unchanged.The paper attributes the unchanged strength to α controlling the ranking-score margin required to reverse a recommendation.
- Complexity and Strength: Removing items at the first recommendation position requires 1.59 more aspects than removing items at the fifth, reflecting larger required changes.User-oriented F1 decreases with complexity but is relatively independent of strength; higher strength yields more explanations that are both necessary and sufficient.
- Complexity and Strength: Explanation complexity and strength are orthogonal: complexity relates to coverage of user preferences, whereas strength relates to the recommendation model’s mechanism.These observations support seeking explanations that are both simple and effective.
7 CONCLUSIONS AND FUTURE WORK
CountER generates explanations through counterfactual changes to item aspects. The paper identifies unexplored extensions to images, textual descriptions, knowledge graphs, and graph neural networks.
- CountER generates explanations based on counterfactual changes to item aspects.
- Counterfactual explainable recommendation remains an early-stage area with substantial room for exploration.
- Future work could apply counterfactual changes to images and textual descriptions.
- CountER's essential idea could extend to explainable decision making over knowledge graphs or graph neural networks.