Source-linked AI summary
Towards Long-term Fairness in Recommendation
Yingqiang Ge, Shuchang Liu, Ruoyuan Gao, Yikun Xian, Yunqi Li, Xiangyu Zhao, Changhua Pei, Fei Sun, Junfeng Ge, Wenwu Ou, Yongfeng Zhang
TL;DR
Fairness-aware recommendation has largely treated popularity groups and fairness constraints as static, despite recommendation feedback changing item popularity and group labels. The paper models recommendation as a CMDP and uses fairness-constrained reinforcement learning to adapt policy decisions over time. Experiments report better recommendation accuracy, short-term fairness, and long-term fairness than baselines.
Problem
Prior fairness-aware recommendation methods use static or one-shot solutions, leaving long-term exposure fairness with popularity-based groups that change through recommendation feedback insufficiently addressed.
Method
The paper models sequential recommendation as a CMDP and applies fairness-constrained reinforcement learning to adjust policy decisions as item exposure and popularity-based group labels change.
Results
Experiments on standard benchmark datasets report better recommendation accuracy, short-term fairness, and long-term fairness than baselines.
Takeaways & Limitations
The framework supports long-term fairness learning for recommendation scenarios in which item popularity and group labels evolve through user interactions.
Takeaways & Limitations
FOE cannot be applied directly to sequential LIRD and therefore reranks only the base ranker's top-200 items before selecting the final top-K list.
Abstract
from arXiv · showhide
As Recommender Systems (RS) influence more and more people in their daily life, the issue of fairness in recommendation is becoming more and more important. Most of the prior approaches to fairness-aware recommendation have been situated in a static or one-shot setting, where the protected groups of items are fixed, and the model provides a one-time fairness solution based on fairness-constrained optimization. This fails to consider the dynamic nature of the recommender systems, where attributes such as item popularity may change over time due to the recommendation policy and user engagement. For example, products that were once popular may become no longer popular, and vice versa. As a result, the system that aims to maintain long-term fairness on the item exposure in different popularity groups must accommodate this change in a timely fashion. Novel to this work, we explore the problem of long-term fairness in recommendation and accomplish the problem through dynamic fairness learning. We focus on the fairness of exposure of items in different groups, while the division of the groups is based on item popularity, which dynamically changes over time in the recommendation process. We tackle this problem by proposing a fairness-constrained reinforcement learning algorithm for recommendation, which models the recommendation problem as a Constrained Markov Decision Process (CMDP), so that the model can dynamically adjust its recommendation policy to make sure the fairness requirement is always satisfied when the environment changes. Experiments on several real-world datasets verify our framework's superiority in terms of recommendation performance, short-term fairness, and long-term fairness.
1 INTRODUCTION
Fairness-aware recommendation must move beyond one-shot solutions because recommendation environments and popularity-based group labels change through user interactions. The paper addresses long-term exposure fairness with dynamically constrained reinforcement learning.
- Existing fairness approaches mainly assess immediate effects in static settings, overlooking long-term consequences.
- Static fairness provides a one-time constrained solution, whereas dynamic fairness adapts to changing environmental factors over time.
- The paper studies item-exposure fairness when popularity-based group labels change because recommendation strategies and user feedback alter exposure.
- It models sequential consumer–recommender interactions as an MDP and imposes dynamic exposure-fairness constraints within a CMDP.
2 RELATED WORK
Prior recommendation-fairness research optimizes utility and fairness under largely fixed attributes or group labels. This paper targets dynamic group fairness, where item labels change through the interactive recommendation process.
- Existing methods also address individual, group, and multi-sided fairness, but their effectiveness is established mainly for fixed recommendation attributes or labels.
- Fairness-aware ranking methods optimize utility under fairness constraints, fairness under utility bounds, or utility and fairness jointly.
- Closest dynamic-fairness studies incorporate feedback to adjust changing item utility but do not address group labels that change during recommendation.
- The paper presents itself as the first attempt to study dynamic group fairness for changing item group labels.
3 PRELIMINARY
The preliminary material defines MDPs and CMDPs as reinforcement-learning frameworks for optimizing discounted reward under transition dynamics and cost constraints. It then introduces CPO's trust-region policy updates.
- Markov Decision Processes: An MDP comprises states, actions, transition probabilities, rewards, an initial-state distribution, and a discount factor, with a policy selecting action distributions.
- Markov Decision Processes: A trajectory records alternating states and actions, with its distribution determined by the policy, initial-state distribution, and transition function.
- Constrained Markov Decision Processes: A CMDP augments an MDP with auxiliary cost functions and limits that restrict allowable policies by bounding discounted total costs.
- Constrained Policy Optimization: CPO replaces policy-divergence penalties with a trust region, enabling larger policy-update steps while constraining average KL divergence.
4 PROBLEM FORMULATION
The paper formulates recommendation as a CMDP with rewards and fairness costs, then defines exposure-based constraints for popularity groups whose labels change over time. FCPO combines these elements to maximize cumulative reward while enforcing fairness during recommendation.
- 4 PROBLEM FORMULATION: The recommendation process is modeled as a CMDP, combining a general constrained formulation with recommendation-specific fairness constraints.The framework first models recommendation with general constraints, then defines fairness constraints and combines them into a fairness-constrained optimization problem.
- 4.1 CMDP for Recommendation: At each timestamp, the agent maps the current user and candidate-item representations to a recommendation list, receives user feedback, and transitions to a new state.The state includes recent user interaction information, while the action is a list of K candidate items.
- 4.1 CMDP for Recommendation: The CMDP assigns immediate reward from feedback and cost from a problem-specific function applied to the recommended list.Feedback may include clicks, skips, or purchases; the cost includes the number of items from a specified group exposed at time t.
- 4.2 Fairness Constraints: Exposure-based demographic parity requires equal average exposure across item groups at each iteration, with groups divided by item popularity.Because recommendation and feedback change exposure, the popularity-based group labels can change during the recommendation process.
- 4.2 Fairness Constraints: Exact-K fairness constrains the proportion of protected candidates in every K-item recommendation list to remain at or below a tunable maximum α.The paper presents this constraint as a practical alternative whose threshold can be adjusted by the recommender system.
- 4.3 FCPO: Fairness Constrained Policy Optimization: FCPO learns a policy that maximizes cumulative reward under a fairness constraint, using an actor and two critics.Figure 1 illustrates the proposed method, and the cost counts exposed items in the sensitive group.
- 4.3 FCPO: Fairness Constrained Policy Optimization: The fairness requirement is imposed at each iteration and converted into a discounted total-cost constraint over a recommendation trajectory.The trajectory length is T, and the discounted cost must satisfy the specified fairness limit.
- 4.3 FCPO: Fairness Constrained Policy Optimization: Because the model has one linear fairness constraint, the constrained optimization problem admits an analytical solution when feasible.The paper notes that this property follows from the single-constraint formulation.
5 PROPOSED FRAMEWORK
The proposed solution follows an Actor-Critic learning scheme augmented with an extra critic for the fairness constraint.
- 5 PROPOSED FRAMEWORK: The framework uses Actor-Critic learning with an additional critic network dedicated to the fairness constraint.The actor and critics are constructed and learned as separate components.
5.1 The Actor
The Actor samples recommendation actions from the current user state, which combines user information and recent interaction history. It then converts the sampled representation into ranked candidate items.
- 5.1 The Actor: The actor π_θ is a stochastic policy that samples a K-item recommendation action from the current user state.The action belongs to the candidate-item space I^K.
- 5.1 The Actor: The state representation concatenates a user embedding with a history embedding derived from recent interactions.The actor architecture includes both the actor network and the state representation model.
- 5.1 The Actor: The history embedding is produced by encoding the user’s most recent N interacted items with a GRU.The interaction history is organized as a queue and updated when recommended items satisfy the paper’s update condition.
- 5.1 The Actor: Updating the history queue preserves a state representation of the user’s most recent interests.The paper explicitly links the queue update mechanism to representing recent interests.
- 5.1 The Actor: An MLP maps the encoded state to the mean and diagonal covariance of a Gaussian action distribution, from which the actor samples a vector.The Gaussian has mean μ and covariance matrix Σ with only diagonal elements nonzero.
- 5.1 The Actor: The sampled representation scores candidate items using the item-embedding matrix and selects items by rank.The candidate-item similarity is computed with a dot product, and P_k,i denotes the probability of selecting item i at rank k.
5.2 The Critics
The framework uses one critic to estimate reward value and a separate critic to estimate cost value for constrained policy optimization.
- 5.2 The Critics: The reward critic V_ω(s_t) approximates the state-value function and is updated with temporal-difference learning by minimizing mean squared error.Its target is y_t = r_t + γrV_ω(s_t+1).
- 5.2 The Critics: A separate critic V_ϕ(s) supports constrained policy optimization by learning from cost-based temporal-difference targets.Its target is y_t = c_t + γcV_ϕ(s_t+1).
5.3 Training Procedure
FCPO training alternates trajectory generation with model updating, using constrained optimization conditions to refine the policy.
- Each training round has trajectory generation and model updating phases, with each trajectory containing T transition results.
- Algorithm 1 trains FCPO using step size δ, cost limit d, and line search ratio β.
- Algorithm 1 uses feedback based on Eq. (8) and Eq. (9) during parameter training.
- The training loop continues until the updated policy remains in the trust region, improves the loss, and satisfies cost ≤ d.
5.4 Testing Procedure
Testing evaluates FCPO in both short-term and long-term settings, with long-term testing updating model parameters and dynamically changing item labels over repeated recommendations.
- After training, FCPO is evaluated on public real-world datasets using separate short-term and long-term evaluations.
- Short-term testing generates trajectories without updating model parameters, then computes recommendation performance and fairness from logged exposure records with fixed group labels.
- Long-term testing updates model parameters online while item labels and fairness constraints change according to current impression results.
- The long-term procedure repeatedly recommends T times, producing T K recommended items in total.
6 EXPERIMENTS
Experiments on MovieLens datasets compare FCPO with traditional, reinforcement-learning, and fairness-aware baselines using accuracy, exposure inequality, popularity, and runtime measures. FCPO achieves stronger recommendation performance and fairness trade-offs, while dynamic policy adjustment improves long-term fairness but can increase fluctuation.
- Experimental Setup: Experiments use MovieLens100K and MovieLens1M, containing one hundred thousand and one million user transactions, respectively.
- Experimental Setup: Items are split by exposure-based popularity: the top 20% form popular group G0, while the remaining 80% form long-tail group G1.
- Baselines: The baselines include MF, BPR-MF, NCF, LIRD, and FOE-based reranking methods.
- Evaluation Metrics: Evaluation covers Recall, F1, NDCG, Gini Index, and Popularity Rate, with lower Gini Index and Popularity Rate considered fairer.
- Recommendation Performance: FCPO achieves the best top-K recommendation performance on both datasets, improving over NCF by 33.09% on Movielens100K and over LIRD by 18.65% on Movielens1M.
- Short-term Fairness Performance: FCPO offers a better trade-off than FOE between NDCG and both individual fairness measured by Gini Index and group fairness measured by Long-tail Rate.
- Runtime: For K=100 recommendations on a single-core CPU, FOE reranking time grows sharply with candidate-set size, while FCPO selects from the entire item set.
- Long-term Fairness in Recommendation: FCPO continuously suppresses fairness metrics during testing, whereas MF-FOE converges quickly and changes mainly after exposure information updates; FCPO also fluctuates more.
7 CONCLUSION AND FUTURE WORK
The paper models long-term fairness in recommendation with dynamically changing group labels and uses a fairness-constrained reinforcement learning framework. Experiments show improved recommendation accuracy, short-term fairness, and long-term fairness.
- The framework models long-term fairness while accounting for dynamically changing group labels.
- A fairness-constrained reinforcement learning framework addresses the dynamic fairness problem in recommendation.
- Experiments on standard benchmark datasets show better recommendation accuracy, short-term fairness, and long-term fairness.