Clicks can be Cheating: Counterfactual Recommendation for Mitigating Clickbait Issue

Wenjie Wang; Fuli Feng; Xiangnan He; Hanwang Zhang; Tat-Seng Chua

Source-linked AI summary

Clicks can be Cheating: Counterfactual Recommendation for Mitigating Clickbait Issue

Wenjie Wang, Fuli Feng, Xiangnan He, Hanwang Zhang, Tat-Seng Chua

arXiv:2009.09945v4Sep 21, 2020cs.IR

TL;DR

Click-based recommendation can promote attractive but low-quality items because clicks do not reliably reflect satisfaction, while post-click feedback is sparse. The paper uses a causal graph and counterfactual inference to remove the direct effect of exposure features, and reports improved satisfaction-oriented recommendation performance. The framework is instantiated on MMGCN but is intended to be model-agnostic, with causal-graph scope and estimation choices remaining boundaries.

Problem
Click-through-rate optimization can emphasize attractive exposure features despite disappointing content, while explicit post-click feedback is much sparser than click data.
Method
The framework models exposure, content, and prediction causally, estimates exposure features' direct effect in a counterfactual world, and subtracts it during inference.
Results
CR outperforms all baselines, including NT by 11.11% and 7.47% w.r.t. N@10 on Tiktok and Adressa, respectively.
Takeaways & Limitations
The model-agnostic CR framework can be adopted across recommendation models and scenarios with minor adjustments while mitigating clickbait.
Takeaways & Limitations
The paper notes that its causal graph is not yet comprehensive and calls for finer-grained causal relations in future work.

Abstract

from arXiv · show

Recommendation is a prevalent and critical service in information systems. To provide personalized suggestions to users, industry players embrace machine learning, more specifically, building predictive models based on the click behavior data. This is known as the Click-Through Rate (CTR) prediction, which has become the gold standard for building personalized recommendation service. However, we argue that there is a significant gap between clicks and user satisfaction -- it is common that a user is "cheated" to click an item by the attractive title/cover of the item. This will severely hurt user's trust on the system if the user finds the actual content of the clicked item disappointing. What's even worse, optimizing CTR models on such flawed data will result in the Matthew Effect, making the seemingly attractive but actually low-quality items be more frequently recommended. In this paper, we formulate the recommendation models as a causal graph that reflects the cause-effect factors in recommendation, and address the clickbait issue by performing counterfactual inference on the causal graph. We imagine a counterfactual world where each item has only exposure features (i.e., the features that the user can see before making a click decision). By estimating the click likelihood of a user in the counterfactual world, we are able to reduce the direct effect of exposure features and eliminate the clickbait issue. Experiments on real-world datasets demonstrate that our method significantly improves the post-click satisfaction of CTR models.

1 INTRODUCTION

The paper identifies a gap between clicks and user satisfaction caused by attractive exposure features, and proposes causal counterfactual inference to mitigate clickbait in recommendation.

Motivation: CTR optimization can favor items whose attractive titles or cover images induce clicks despite disappointing actual content.Such optimization may make seemingly attractive but low-quality items recommended more frequently.
Motivation: Clicks and post-click satisfaction can diverge when exposure features mislead users about an item's content.The paper illustrates this contrast using clicks followed by likes or dislikes.
Problem: Post-click likes and dislikes are a direct feedback source, but they are much rarer than click data in real-world datasets.This scarcity makes explicit feedback difficult to use alongside large-scale implicit feedback.
Approach: The proposed method builds a causal graph and estimates exposure-feature effects in a counterfactual world containing only what users can see before clicking.Inference removes this direct exposure effect from the factual prediction score.
Contributions: The framework introduces counterfactual inference into recommendation and is implemented on MMGCN with experiments on two benchmarks.The contributions describe the framework as applicable to recommender models using item features as inputs.

2 TASK FORMULATION

This section formalizes click-based recommender training, the clickbait ranking failure, and evaluation using clicks that receive positive post-click feedback.

Recommender training: Recommender models learn a scoring function from user and item features, with item features divided into exposure and content features.Exposure features are observed before clicks, while content features are observed after clicks; training typically uses historical click data.
Recommender training: During inference, items are ranked by predicted preference and the highest-ranked items are recommended.The scoring function is trained using a recommendation loss such as cross-entropy.
Clickbait issue: The clickbait issue occurs when an attractive-exposure, dissatisfying-content item ranks above a less attractive-exposure, satisfying-content item.This ranking displaces items with satisfying content and can produce clicks ending in dislikes.
Clickbait issue: Clickbait can create a vicious spiral in which click-based training further aggravates the issue over time.The paper aims to break this spiral during inference by favoring satisfaction rather than higher CTR.
Evaluation: Evaluation counts only testing-period clicks followed by positive post-click feedback, excluding clicks whose satisfaction is unknown.This differs from conventional evaluation that treats all testing-period clicks as positive samples.

3 PRELIMINARY

The preliminary section introduces causal graphs, counterfactual inference, and causal-effect decompositions used to separate direct and indirect effects.

Causal Graph: A causal graph is a directed acyclic graph whose nodes represent variables and edges represent causal relations.Structural equations describe how variables are affected by their parents.
Counterfactuals: Counterfactual inference estimates descendant outcomes under a different treatment value while retaining the factual values of other variables.The do-operator externally intervenes on a variable in the structural equations.
Counterfactuals: In the example, counterfactual inference keeps education at its factual value on the direct path while setting skill to the reference value.This represents the income outcome if skill matched that of a person without qualifications.
Causal Effect: Total effect measures the response change when treatment moves from a reference value to an expected value.The paper decomposes total effect as TE = NDE + TIE, separating natural direct and total indirect effects.
Causal Effect: The total indirect effect changes mediators from reference values to expected values while keeping the treatment fixed on the direct path.It can be obtained by subtracting the natural direct effect from the total effect.

4 COUNTERFACTUAL RECOMMENDATION

The CR framework models exposure features' direct effect on recommendation scores and uses counterfactual inference to reduce clickbait-driven rankings. It combines a causal graph, multi-task training, and counterfactual inference while adapting existing recommender models through feature fusion.

Causal graph: CTR training can emphasize exposure features over content features because exposure features directly cause clicks on clickbait items.The resulting mismatch between the training-data generation process and the conventional graph can increase promotion of attractive but low-quality items.
Causal graph: The proposed causal graph adds a direct exposure-feature path E→Y alongside the indirect path E→I→Y.This represents both direct and mediated effects of exposure features on prediction.
Counterfactual inference: NDE estimates the prediction effect of changing exposure features while holding the mediated item representation at reference values.The counterfactual asks how strongly a user would be attracted by exposure features alone, with reference exposure and content values treated as unavailable features.
Counterfactual inference: CR subtracts the exposure-feature NDE from the real-world prediction, suppressing items whose attractiveness comes mainly from exposure features.The framework therefore relies more on combined item-feature effects during ranking; CR can be expressed as Y_CR = TE − NDE.
CR framework design: The framework upgrades the scoring function to accept user, item, and exposure features, using late fusion to retain compatibility with existing recommender models.The fusion function combines predictions from models using user-item features and user-exposure features; multiplication is one adopted strategy.
CR framework design: CR differs from conventional recommendation through its causal graph, counterfactual-world multi-task training, and NDE-based inference.The framework is designed to be applied to recommender models that use item features as inputs.

5 RELATED WORK

Related work addresses recommendation bias and the click–satisfaction gap through feedback incorporation, negative-experience identification, and causal debiasing. These studies span multimodal and graph-based recommendation, implicit-feedback refinement, and causal approaches to logged-data bias.

Recommendation: Multimodal recommendation incorporates rich user and item features, while GNN-based MMGCN uses modality-aware propagation over the user-item graph.Existing approaches are trained by implicit feedback, according to the supplied passage.
Incorporating Various Feedback: Clickbait mitigation methods either identify negative interactions before training or directly incorporate post-click feedback such as ratings, favorites, and dwell time.The first category uses a two-stage pipeline that retains interactions identified as positive.
Causal Recommendation: Causal recommendation research applies causal inference to fairness and logged-data biases, including position, exposure, and popularity bias.Inverse propensity scoring is described as a popular debiasing method within this research area.

6 EXPERIMENTS

Experiments on Tiktok and Adressa evaluate counterfactual recommendation against baselines using post-click feedback and ranking metrics. CR consistently improves recommendation performance, reduces exposure-feature-driven promotion of low-satisfaction items, and gains more as click data becomes less clean.

Experimental Settings: Experiments use Tiktok and Adressa, with post-click feedback used to construct evaluation sets while click interactions provide training data.Tiktok treats thumbs-up, favorite, or finish as positive feedback; Adressa treats clicks with dwell time over 30 seconds as likes.
Experimental Settings: Evaluation ranks all candidate items for each user and reports P@K, R@K, and N@K at K={10,20}, where higher values are better.
Performance Comparison: Discarding exposure features or using inverse propensity weighting does not reliably mitigate clickbait, while post-click-feedback methods can suffer when feedback is sparse.On Tiktok, CT's NDCG@10 decreases by 11.71% relative to NT, and clicks ending with likes account for only 39.44%.
Performance Comparison: CR achieves significant gains over all baselines, improving N@10 over NT by 11.11% on Tiktok and 7.47% on Adressa.CR also outperforms RR, which uses post-click feedback to rerank recommendations.
In-depth Analysis: CR recommends fewer low like/click-ratio items, produces flatter recommendation-frequency distributions, and usually increases rank_gap relative to NT.
In-depth Analysis: CR outperforms NT across filtered datasets, with performance gains increasing sharply when the discarded proportion exceeds 0.4.This indicates greater gains in scenarios containing more clicks that end with dislikes.
In-depth Analysis: Non-linear fusion strategies outperform linear ones, and SUM-tanh achieves the best performance among the tested fusion strategies.The results suggest that fusion-function boundaries affect CR performance across datasets.
In-depth Analysis: Synthetic tests pair real items with identical content features but deceptive exposure features from items with like/click ratio below 0.5.A model mitigating clickbait should rank each fake item below its paired real item.

7 CONCLUSION AND FUTURE WORK

CR removes the estimated direct effect of exposure features through counterfactual inference, mitigating clickbait while accounting for causal relations in recommendation. Although instantiated on MMGCN, the framework is model-agnostic and supports broader adoption; future work calls for richer causal graphs and other click-bias analyses.

CR models causal relations among exposure features, content features, and predictions, then removes exposure features' estimated direct effect from recommendation scores.
Although instantiated on MMGCN, CR requires only minor adjustments to be adopted across other recommendation models and scenarios.
The work motivates incorporating counterfactual inference into recommender systems and exploring more comprehensive causal graphs with finer-grained relations.
Future research should examine other intrinsic click-data biases, including selection bias and position bias, alongside additional causal-inference techniques.