Source-linked AI summary
Why This, Not That? Mining User Profiles for Pair-wise Counterfactuals
Meysam Varasteh, Veronika Bogina, Noam Koenigstein, Robin Burke
TL;DR
The paper addresses the limited focus of recommender explanations on individual items by asking why one recommended item ranks above another. It develops counterfactual learning methods that identify profile items contributing to relative rankings, and finds that comparative objectives with a Siamese network perform particularly well across multiple datasets and algorithms. The authors also identify practical limitations involving impossible reversals and popularity effects.
Problem
Recommender explanations have largely focused on single items, leaving the comparative question of why item A ranks higher than item B less addressed.
Method
The paper extends LXR with counterfactual comparative explainers that learn profile masks identifying items whose removal could reverse the target and comparative items’ rankings.
Results
A combined comparative objective with a Siamese network performs best, while counterfactual profile perturbations effectively generate contrastive ranking explanations and explicitly optimizing comparative explanations improves performance.
Takeaways & Limitations
Comparative ranking explanations can be grounded in the recommender’s operation by identifying user-profile items associated with relative item rankings.
Takeaways & Limitations
Some rank reversals may be impossible because of strong popularity effects, and removing highly popular profile items may produce explanations that are counterfactually correct but unintuitive.
Abstract
from arXiv · showhide
The topic of explanation in recommender systems has seen steady research attention since the earliest days of the field. With some exceptions, this work has focused on the explanation of single items in a recommendation list and, especially recently, has emphasized approaches that are decoupled from the logic of the recommendation algorithm itself. Based on findings in the psychology of interpersonal communication, we propose a new task, pairwise interpretation of item rankings, asking the comparative question ``Why is item A ranked higher than item B?''. An effective solution to this task, we argue, is inherently grounded in the operation of the recommendation algorithm. We propose a class of techniques based on counterfactual learning to uncover the items in a user's profile that have contributed to the relative ranking of items. Using multiple datasets, we show that it is possible to identify such items as potential basis for comparative explanation.
1. Introduction
The paper reframes recommender explanations from asking why one item was recommended to asking why a target item ranks above a comparative item. It proposes counterfactual, algorithm-grounded explanations that identify profile items whose removal would reverse the ranking.
- The paper studies why target item T is ranked higher than comparative item C, extending beyond explanations of individual recommended items.
- Comparative explanations focus on factors distinguishing two competing items, which may make them more intuitive and relatable than listing isolated reasons for recommending T.
- The ranking question is counterfactual because it asks which minimal profile changes would reverse the relative positions of T and C.
- The paper uses counterfactual learning to identify profile items whose removal would reverse the ranking, calling such a set a perturbation.
- Explanation quality is framed in terms of veracity, requiring an actual rank reversal, and clarity, favoring the smallest effective perturbation.
- The study evaluates whether counterfactual learning can efficiently approximate perturbations and examines contrastive objectives across datasets and recommendation algorithms.
2. Related Work
Prior recommender explanations largely address single items, while related counterfactual and comparative methods either search for input changes, learn perturbations at scale, or compare item attributes. This paper targets comparative explanations for two recommended items and adapts a learned profile-mask framework.
- Recommender explanation research has primarily developed techniques for explaining the presence of a single item in a ranked list.
- Model-agnostic methods such as SHAP and LIME perturb inputs to estimate feature contributions across machine-learning models, including recommenders.
- Counterfactual recommendation explanations seek input alterations that change predictions, but individual perturbation searches can be prohibitive in practical settings.
- Learning-based approaches predict profile or feature masks at scale instead of searching for perturbations separately for every instance.
- Other comparative explanation studies compare items through attributes, while ranking-oriented work in information retrieval asks why one document is ranked above another.
- Comparing two recommended items counterfactually is distinct from comparing a recommendation with items already present in the user’s profile.
3. Explanation Models
The paper extends the learning-based LXR framework from single-item explanations to comparative explanations using profile masks for a target and comparative item. It presents score-based and directly optimized variants, including a Siamese-network objective and TDLR-based training.
- LXR: LXR learns a mask over a user’s profile to identify items whose removal would make a recommended item leave the top recommendation slate.
- LXR: A profile perturbation Π consists of removed profile items, with veracity requiring the target recommendation to disappear and conciseness favoring a small effective subset.
- Comparative Explanation: The comparative explainer takes user history, target item, and comparative item as inputs and outputs a mask estimating profile-item contributions to the target’s preference over the comparative item.
- Comparative Explanation: CLXR-score combines independently generated target and comparative explanation maps, emphasizing items that reduce the target’s rank without reducing the comparative item’s rank.
- Learning Comparative Explanations: The objective combines comparative, prediction, inverse, and regularization terms, with regularization promoting sparse masks and the recommender remaining frozen during optimization.
- Implementations: CLXR-joint uses a Siamese network and directly optimizes comparative explanation losses, unlike CLXR-score’s reuse of an explainer trained for single-item explanations.
- Implementations: CLXR-tdlr directly learns the comparative mask using a Targeted Deviation–Logit Ranking loss, removes prediction and inverse losses, and retains an ℓ1 sparsity constraint.
4. Evaluation Metrics
The paper evaluates comparative explanations by measuring how much of a user profile must be removed to reverse two items’ ranks and how often such reversals can be found. These metrics expose a trade-off between succinctness and coverage, while some reversals may be impossible because of popularity effects.
- Evaluation setup: The evaluation focuses on the top-ranked item y_t and a comparative item y_c sampled from the other top-10 recommendations.The comparative explanation task is novel, so the authors define metrics around this restricted set of item pairs.
- Perturbation metrics: MPRR measures the smallest profile perturbation that reverses the ranks of y_t and y_c, with smaller values indicating a better explainer.Perturbations remove progressively larger fractions of profile items ordered by their attribution scores.
- Perturbation metrics: MPRR% normalizes perturbation size by profile length, while MPRR# reports the corresponding absolute number of removed items.The percentage makes perturbation sizes more comparable across datasets.
- Metric trade-off: Popularity effects can make some rank reversals fundamentally impossible, requiring a fallback to general, non-personalized explanations.The paper illustrates this boundary with niche and blockbuster movie recommendations.
- Perturbation metrics: Coverage reports the proportion of users for whom a rank-reversing perturbation can be generated.Users without a discovered reversal are assigned an infinite MPRR and excluded from the MPRR calculation.
- Metric trade-off: High Coverage and low MPRR# represent competing goals because broader coverage may require accepting larger perturbations.The appropriate balance depends on the intended application.
5. Methodology
The methodology trains and evaluates comparative explainers across three datasets and recommendation models, using user profiles, ranked target–comparator pairs, and attribution masks. It compares learned and baseline explanation approaches while examining how the recommendation algorithm affects performance.
- Datasets and recommendation models: Experiments use ML-1M, Yahoo Music, and Pinterest datasets, with preprocessing that retains sufficiently rated users and items.For ML-1M, implicit ratings retain ratings of 3.5 or higher, producing 575,128 ratings from 6,037 users across 3,381 items.
- Datasets and recommendation models: The explainer is designed to be algorithm-agnostic, but its performance depends on the recommendation algorithm, particularly through popularity bias.The experiments therefore report results for two recommendation models across all three datasets.
- Datasets and recommendation models: The study evaluates matrix factorization and variational autoencoder recommendation models, alongside a popularity-ranking baseline.Matrix factorization represents users and items with latent vectors, while VAE uses encoder–decoder representations.
- Explanation methods: The compared explanation methods include CLXR variants, LXR, SHAP, LIME, and ACCENT, spanning comparative, target-focused, and model-agnostic approaches.SHAP is adapted by grouping each user’s items into 10 clusters because exhaustive perturbation is computationally expensive.
- Explainer training: Recommendation algorithms generate slates in which the rank-1 item is the target and a random item from ranks 2–10 is the comparator.These target–comparator tuples form the explainer’s training data.
- Explainer evaluation: During evaluation, the comparative explainer assigns profile-item contributions through a mask, then removes increasingly large top-attribution subsets to recompute rankings.The resulting perturbed recommendations are used to calculate MPRR% and Coverage.
6. Results
Across datasets and recommendation models, CLXR-joint and CLXR-tdlr are generally the strongest explainers, with ACCENT best on VAE/Pinterest. Popularity filtering reduces coverage, suggesting popular items often contribute to perturbations but may produce less convincing explanations.
- CLXR-joint consistently outperforms baselines in Coverage and MPRR except on the VAE/Pinterest task, where ACCENT performs best.CLXR-tdlr is also competitive, particularly with Matrix Factorization.
- CLXR-joint and CLXR-tdlr dominate most cumulative Coverage-versus-perturbation plots, except that ACCENT outperforms them for VAE/Pinterest.The plots emphasize small perturbations: below 50 items for MovieLens and below 25 for Yahoo and Pinterest.
- For Pinterest with Matrix Factorization, CLXR-joint and CLXR-tdlr perform similarly and remain the dominant methods.
- In the Alien–Braveheart example, removing Halloween, Forbidden Planet, and Brazil reverses the recommendation ranking.The perturbation is generated by CLXR-joint trained with a VAE recommender on ML-1M.
- The Impact of Popularity: Removing the top 15% most popular items from profiles lowers Coverage, while MPRR increases slightly in most conditions.The recommendation model remains unchanged, and the results suggest popular items form a large part of many CLXR-joint perturbations.
- The Impact of Popularity: Popularity-filtered explanations can still be generated without popular items, but their perceived quality requires further study with human subjects.
7. Conclusion and Future Work
The paper develops counterfactual comparative explanations for item rankings and finds that contrastively optimized models can produce focused, faithful perturbation masks. Results are broadly consistent across datasets, while highlighting limits from large perturbations, target-position bias, and the need for user-centered evaluation.
- Contributions: A combined contrastive objective with a Siamese network provides the best performance among the comparative explanation techniques evaluated.The approach extends LXR counterfactual explanation to compare a target item with a comparative item.
- Contributions: Counterfactual profile perturbations effectively generate contrastive explanations, although their efficiency does not guarantee truly minimal explanations.The authors identify the gap between efficient approximations and minimal perturbations as an open challenge.
- Contributions: Explicitly optimizing for the target–comparative preference improves performance, producing more focused and faithful masks than methods without a contrastive loss.CLXR-joint jointly optimizes the preference between the two items.
- Generalizability: Experiments across multiple benchmark datasets show consistent overall trends alongside dataset-specific behaviors, including stronger popularity effects for frequently top-ranked targets.These findings motivate further study of how target rank and recommendation algorithms influence explanation behavior.
- Future work: Technical offline evaluation should be complemented by human-centered studies of whether explanations are comprehensible, actionable, and aligned with users’ comparative questions.The authors plan a follow-up user study addressing these qualities.
- Future work: Counterfactual perturbations can sometimes affect dozens of profile items, motivating clustering or other methods that summarize larger perturbations.The paper illustrates this scope boundary with an example involving 25 Marvel action films.