PaperlayerView exact revision on arXiv

Source-linked AI summary

Estimating and Penalizing Induced Preference Shifts in Recommender Systems

Micah Carroll, Anca Dragan, Stuart Russell, Dylan Hadfield-Menell

arXiv:2204.11966v2cs.LGcs.IR

TL;DR

Recommenders can alter user preferences, while engagement metrics do not directly value those shifts. The paper learns preference dynamics from interaction histories and evaluates shifts against safe-shift metrics; in simulation, the metric rates the RL policy as 35% worse than the myopic one, but translation to real users remains uncertain.

  • Problem

    Long-term engagement incentives can favor easier-to-satisfy preferences, while standard engagement metrics do not directly value preference shifts.

    Source: §1 · pp. 1–2

  • Method

    The authors learn preference dynamics from historical interactions and compare policy-induced shifts with safe shifts, including shifts under a random recommender.

    Source: §1 · p. 2, §4.2 · p. 4

  • Results

    The metric evaluates the RL policy from Fig. 1 as 35% worse than the myopic one, and the learned model anticipates shifts almost as well as knowing the true dynamics.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Safe-shift metrics can evaluate preference changes and serve as optimization proxies, though blindly minimizing deviation could encourage static preferences.

    Source: §5 · p. 5, §8 · p. 8

    Experiments use simulated users because ground-truth preferences are needed, and the authors cannot guarantee results will transfer to real users with more complex dynamics.

    Source: §8 · p. 8

Abstract

from arXiv · show

The content that a recommender system (RS) shows to users influences them. Therefore, when choosing a recommender to deploy, one is implicitly also choosing to induce specific internal states in users. Even more, systems trained via long-horizon optimization will have direct incentives to manipulate users: in this work, we focus on the incentive to shift user preferences so they are easier to satisfy. We argue that - before deployment - system designers should: estimate the shifts a recommender would induce; evaluate whether such shifts would be undesirable; and perhaps even actively optimize to avoid problematic shifts. These steps involve two challenging ingredients: estimation requires anticipating how hypothetical algorithms would influence user preferences if deployed - we do this by using historical user interaction data to train a predictive user model which implicitly contains their preference dynamics; evaluation and optimization additionally require metrics to assess whether such influences are manipulative or otherwise unwanted - we use the notion of "safe shifts", that define a trust region within which behavior is safe: for instance, the natural way in which users would shift without interference from the system could be deemed "safe". In simulated experiments, we show that our learned preference dynamics model is effective in estimating user preferences and how they would respond to new recommenders. Additionally, we show that recommenders that optimize for staying in the trust region can avoid manipulative behaviors while still generating engagement.

1. Introduction

Recommenders can shape user preferences, and long-horizon engagement optimization may incentivize systems to shift preferences toward states that are easier to satisfy. The paper proposes anticipating these shifts and evaluating them against safe reference shifts before deployment.

Motivation

  • Long-horizon engagement policies may have incentives to manipulate preferences, while even myopic systems can systematically induce undesirable changes.Preferences that are easier to satisfy can create more engagement opportunities, for example when more content is available for them or stronger preferences produce higher engagement.Source: §1 · p. 1

Contribution

  • The paper proposes learning from historical interactions to anticipate how user preferences would evolve under a new recommender policy.Its Figure 1 contrasts preference trajectories under RL and myopic policies with natural evolution, with RL driving preferences to one spot and the myopic policy doing so to a lesser degree.Source: §1 · p. 2, Figure 1 · p. 2

Approach and limitation

  • Safe-shift metrics compare policy-induced changes with trusted reference shifts, including natural preference evolution under a random recommender.The safe-shift lists are non-exhaustive, so policies that may actually be safe can be conservatively penalized.Source: §1 · p. 2

Results

  • In simulation, the metric rates the RL policy 35% worse than the myopic policy, which is 40% worse than the policy penalized for preference manipulation.The learned metric also correlates with evaluation using ground-truth preference dynamics, and optimizing for safe shifts yields higher-scoring policies.Source: §1 · p. 2

Implications

  • The authors argue that ethical use of recommenders at scale requires measuring and penalizing how systems shift users’ internal states.They present a framework for specifying safe shifts and using them to detect and control incentives to alter preferences.Source: §1 · p. 2

2. Related Work

Prior work estimates recommender effects using static interaction data, hand-crafted user-dynamics simulations, or interventions with real users. This paper instead learns dynamics from data and frames safe shifts in relation to side-effect baselines.

RS effects on users’ internal states

  • Earlier quantitative approaches analyze static datasets, simulate users with hand-crafted dynamics, or estimate effects through interventions with real users.The paper positions its approach as an improvement on hand-specified dynamics, while retaining an assumption of a known choice model.Source: §2 · p. 3

Neural networks for recommendation and human modeling

  • Data-driven human models have been used for click prediction and RL simulation, but the authors report no prior use for explicitly simulating and quantifying hypothetical recommenders’ effects on preferences.Their method also anticipates effects on user behavior.Source: §2 · p. 3

Side effects and safe shifts

  • Safe shifts correspond to the choice of baseline in side-effect research, where reward may omit the costs or value of preference changes.The paper frames the question as how to prevent unwanted preference-shift side effects.Source: §2 · p. 3

3. Preliminaries

The setup represents a user’s preferences as time-indexed values over content and models interaction as a sequence of slates, choices, and state updates. It assumes a known choice model while leaving internal-state dynamics unknown.

Setup

  • A user’s time-indexed preference vector assigns a scalar value to each content item, and the internal state includes preferences plus other variables.The preference vector is not assumed to capture the full internal state, which is described as a sufficient statistic for long-term behavior.Source: §3 · p. 3
  • At each timestep, a policy maps interaction history to a slate, the user chooses an item, and the internal state updates.The policy generates each new slate from the history of previous slates and choices.Source: §3 · p. 3

User choice model and preference evolution

  • The model assumes the form of the user’s choice distribution given the current slate and preferences is known, while internal-state dynamics are treated as unknown.The paper also tests performance under a misspecified choice model.Source: §3 · p. 3

4. Anticipating Users’ Preferences Changes

The method estimates how a new policy would change preferences by extending each user’s historical interaction trajectory. It describes an oracle inference procedure for known dynamics and a learned predictive model for unknown dynamics.

4. Anticipating Users’ Preferences Changes

  • For each user with historical interactions under an existing policy, the method estimates future preferences under a new policy and aggregates estimates across users.The target preferences are those expected after further interactions with the new policy.Source: §4 · p. 3

4.1. Estimation under known user dynamics

  • With known internal-state dynamics, exact inference over future preferences under the new policy can be performed by extending HMM prediction from the interaction history.This is presented as an oracle approach; the future preferences can then be recovered from the inferred state.Source: §4.1 · pp. 3–4

4.2. Estimation under unknown user dynamics

  • When dynamics are unknown, a model learns preference beliefs from historical slates and choices so that, with the assumed choice model, they predict users’ observed choices.The model’s preference beliefs and a new slate induce a choice distribution; simulated trajectories then estimate future preferences under a new policy. Generalizing to new-policy histories is more plausible when training data span diverse policies.Source: §4.2 · p. 4, Figure 2 · p. 4

5. Evaluating Preferences and Avoiding Unwanted Shifts

The section proposes evaluating preference shifts by comparing them with shifts designers treat as safe, then using those metrics to penalize recommenders for deviations. Random-policy-induced natural preference shifts provide one proposed safe baseline.

Safe-shift metrics

  • Safe-shift metrics flag policy-induced preference trajectories that deviate from trusted shifts, supporting both policy evaluation and optimization against potentially unwanted changes.The framework is risk-averse: it identifies shifts warranting investigation rather than defining all desirable changes.Source: §5 · pp. 4–5

Safe shifts: initial preferences

  • Initial preferences offer a crude safe-shift baseline that flags any deviation from users’ starting preferences as potentially problematic.This baseline can classify routine preference changes as concerning because it treats the initial state as the safe reference.Source: §5 · p. 5

Safe shifts: natural preference shifts (NPS)

  • Natural preference shifts use preferences under a random recommender as an approximation of how users might change without the recommender policy shaping their slate exposure.The idealized reference is an omniscient user with full agency over preference evolution, but such users cannot be observed; random slates approximate that reference.Source: §5 · pp. 4–5

Penalized objective

  • Weighted safe-shift distances can be added to long-term engagement objectives to hedge across definitions of value rather than treating preference shifts as cost-free.The resulting objective can be optimized myopically or with long-horizon reinforcement learning.Source: §5 · p. 5

6. Experimental Setup

The experiments use a simulated environment with ground-truth user preferences to test preference estimation and recommender evaluation. Historical interaction data trains the human model, while the simulator permits comparison against true user dynamics.

Simulation rationale

  • The simulator tests recommenders and internal-preference metrics against ground-truth users, while the learned human model uses only observable slates and choices.This setup permits direct evaluation of preference recovery, which real interactions cannot provide because users’ true preferences are unobserved.Source: §6 · p. 5

Ground truth user dynamics

  • At each timestep, a slate drives item choice and engagement, while current preferences and beliefs about future slates inform the distribution over next-timestep preferences.The assumed choice rule is a conditional logit model, and the ground-truth preference dynamics are adapted from Bernheim et al. (2021).Source: §6 · pp. 5–6, Figure 4 · p. 6

Simulated environment setup

  • The simulated cohort starts near preference 130°, with choice-stochasticity temperatures making 80° and 270° attractor points where users are less stochastic.These attractors create incentives for recommenders to shift users toward preferences associated with higher engagement.Source: §6 · p. 6

Dataset

  • The experiments use 10,000 interaction trajectories of length 10, split into 7,500 training and 2,500 validation trajectories.The trajectories were collected under a mixture of policies.Source: §6 · p. 6

Training human models and penalized RS policies

  • A bidirectional transformer models users from historical slates and choices, while PPO trains myopic and RL recommenders over six possible slate distributions.The three metrics used for penalized training receive equal weight.Source: §6 · p. 6

Validation

  • Preference prediction performs similarly to oracle NHMM estimation on held-out trajectories, both with the correct choice model and with some model mis-specification.Figure 5 reports validation losses and accuracies averaged across timesteps.Source: Figure 5 · p. 6

Imagining Natural Preference Shifts

  • The natural-shift experiment compares shifts among 1,000 users with a Monte Carlo estimate from 1,000 simulated trajectories under a random recommender.The simulated trajectories use the learned model and Algorithm 1.Source: Figure 6 · p. 6

7. Results

The experiments test preference estimation under known and alternate recommender policies, then assess safe-shift metrics and penalized training. Estimated evaluations preserve policy rankings, while penalized RL stays closer to safe shifts at some engagement cost.

7.1. Estimating user preferences

  • Choice-prediction validation loss indicates the degree of choice-model misspecification; reducing misspecification reduces that loss.Choice-prediction validation loss is observable in real-world experiments, unlike preference prediction loss and accuracy.Source: §7.1 · p. 7
  • Under alternate policies absent from training data, the learned model estimates population-level preference evolution under a random recommender, capturing natural preference shifts.The simulation uses no past interaction context when estimating these trajectories.Source: §7.1 · p. 7

7.2. Evaluating undesirable shifts and penalizing them

  • In simulation, unpenalized RL drives preferences strongly away from their initial state and concentrates them in one region; the myopic policy shows the same effect less strongly.The authors describe the RL behavior as potentially concerning.Source: §7.2 · p. 7
  • The evaluation sums engagement measured at initial preferences and natural preference shifts as safe-shift proxies, alongside current-preference engagement.The authors treat the sum as a better proxy for true value than any one metric alone.Source: §7.2 · p. 7
  • Validation is confounded because learned user-model quality affects evaluation of the preference-shift metrics themselves.Estimated evaluation and simulation-based training both rely on the learned model of preference dynamics.Source: §7.2 · p. 7
  • Unpenalized RL yields higher engagement than myopic recommendation (7.49 vs 5.71) but performs worse on safe-shift metrics, matching its more undesirable shifts.The comparison uses oracle evaluation and relates to the preference shifts seen in Figure 1.Source: §7.2 · p. 7
  • With oracle training, penalized RL substantially increases cumulative safe-shift metrics at a slight expense of instantaneous engagement and produces shifts closer to safe ones.The authors report that these shifts are drastically better at avoiding unwanted changes overall.Source: §7.2 · p. 8
  • Under learned dynamics, estimated evaluations preserve the oracle policy ranking, and simulated penalized RL has higher oracle-evaluated cumulative reward than unpenalized RL.Estimated evaluation is more optimistic about unpenalized reward, but keeps the same ranking.Source: §7.2 · p. 8

8. Discussion

The discussion presents predeployment preference-shift estimation, safe-shift evaluation, and penalized optimization as contributions, while emphasizing the idealized scope of the experiments. The notation defines the policy-conditioned choices, slates, preferences, and engagement quantities used in the paper.

8. Discussion

  • The paper contributes predeployment estimates of policy-induced preference shifts, a safe-shift evaluation framework, and metrics for penalizing unwanted shifts during optimization.The authors describe the work as a first step tested in an idealized setting with relatively strong assumptions.Source: §8 · p. 8
  • Across 1,000 trajectories, Table 1 reports that penalized systems stay significantly closer to safe shifts than unpenalized systems.The caption reports standard errors below 0.1.Source: Table 1 · p. 8
  • Experiments use simulation because validating the method and metrics requires ground-truth preferences, and transfer to real users is not guaranteed.Real users may have more complex dynamics, potentially requiring more and more diverse data.Source: §8 · p. 8
  • Table 2 reports that estimated evaluations strongly correlate with oracle evaluations and preserve policy rankings; simulation training has a similar effect to on-policy training.The caption describes the comparison as estimating evaluations and simulating training for long-term-value systems.Source: Table 2 · p. 8

A. Notation

  • The notation represents recommender policies as mappings from interaction histories to slates, with policy-indexed user choices and preferences used to define engagement.It also defines policy-conditioned beliefs over future preferences and choices.Source: A. Notation · p. 14

B. Known choice model assumption

The method assumes a known choice model because inferring preferences and their dynamics without one is difficult. It estimates counterfactual preferences by recovering initial preferences from observed interactions and using them to predict trajectories under an alternative policy.

B. Known choice model assumption

  • Inferring preferences without a known choice model is difficult; Baum-Welch-style joint learning would require linear hidden-state dynamics and a known state dimension.The authors expect user dynamics to be nonlinear and do not want to assume the internal-state dimension is known.Source: B. Known choice model assumption · p. 14

C. Counterfactual Preferences Estimation

  • Counterfactual estimation addresses the limits of forward prediction by asking how a user’s preferences would have evolved under an alternative policy from the start.The approach estimates an initial-preference belief from observed interactions, then conditions a counterfactual estimator on that belief.Source: C. Counterfactual Preferences Estimation · pp. 15–16

D. Casting our problem to a Non-Homogeneous Markov Model

  • In the NHMM formulation, observations combine the recommender’s slate and the user’s choice, while hidden-state dynamics depend on the slate and policy history.For a deterministic policy, conditioning on the current slate is equivalent to conditioning on the full interaction history.Source: D. Casting our problem to a… · p. 16

Future preference estimation with known dynamics

  • Future preference estimation predicts preferences under a new policy by propagating beliefs forward, with simulations approximating the NHMM prediction task.The future policy’s history-dependent actions make the forward dynamics time-dependent.Source: Future preference estimation with… · p. 17

Initial preference prediction model output correction

  • Recovering initial preferences from later interactions yields a smoothing estimate, but counterfactual inference retains only initial-preference beliefs rather than the full internal state, reducing accuracy.The authors describe this as the best available without further assumptions.Source: Initial preference prediction model… · pp. 17–18

Preference estimation model form

  • The three preference estimators can use sequence models; the implementation uses transformers and represents multimodal preference beliefs as weighted mixtures of Von Mises distributions.The network outputs distribution parameters and weights for the mixture.Source: Preference estimation model form · pp. 18–19

F. Computing metrics

The paper uses estimated future or counterfactual preferences to compute metrics and train recommenders with penalties. Its simulated setup models preference updates toward more convenient preferences and examines learned human models under choice-model misspecification.

F. Computing metrics

  • Metrics for future preferences use future estimation, while penalized RL compares simulated trajectories with counterfactual preferences under a safe policy.The counterfactual estimate supplies the safe-policy reference for the user and current trajectory.Source: F. Computing metrics · p. 19

The underlying MDP

  • The recommender is modeled as a POMDP with hidden user state, slate actions, and expected per-timestep reward, with proxy terms added for penalized training.The belief-MDP is approximated using deep reinforcement learning with interaction histories as policy inputs.Source: The underlying MDP · p. 19

Penalized RL training

  • Penalized RL training simulates user trajectories with the learned human models, computes penalty metrics, and optimizes the policy on on-policy rollouts.The training procedure includes current-policy and safe-policy preference estimates.Source: Penalized RL training · p. 19

Reduction of distances to final penalized objective

  • The penalized objective weights engagement under safe-policy preferences relative to engagement under the main policy, with the weights treated as hyperparameters.The supplied objective passage identifies two safe-policy preference terms but does not provide their full definitions.Source: Reduction of distances to final… · pp. 19–20

H.1. Ground truth users

  • The simulated choice model weights content features by preference alignment and by their proportion in the slate.When slates are discrete, the proportionality term becomes an indicator for whether an item is present, recovering the traditional logit form.Source: §H.1 · p. 20

Feed belief update

  • Users’ preference updates favor more convenient preferences, and simulated policies can shift preferences toward different engagement modes, which need not reflect user value.Preferences naturally tend toward one mode, while some recommender policies drive them toward the other; the higher engagement optimum is not necessarily desirable.Source: Feed belief update · p. 20

H.2. Learned human models

  • The learned human models use a BERT-style transformer; experiments also vary the choice-model temperature across preference space to test misspecification.Increasing misspecification further led to worse results.Source: §H.2 · p. 21

H.3. Simulated dataset

The simulated historical dataset mixes randomized and goal-directed slate selection, with content features distributed uniformly.

H.3. Simulated dataset

  • Historical interactions use uniform content features and a mixed policy: half the rollouts select among focused slates, while the other half mostly use uniform slates.The second policy chooses a uniform slate 80% of the time and a focused slate 20% of the time, encouraging generalization to a related but different safe policy.Source: §H.3 · p. 21

H.4. RS training

The recommender policies are trained with PPO, using recurrent policies and distinct discount factors for myopic and long-horizon training.

H.4. RS training

  • PPO training uses recurrent policies, with γ = 0 for myopic training and γ = 0.99 for long-horizon RL training.Training runs take less than 30 minutes per condition on a 2020 MacBook Pro 16-inch.Source: §H.4 · p. 22

I. Results

The initial preference predictor performed similarly to oracle estimates across correct and mis-specified choice models, while counterfactual preference estimation was generally worse.

I. Results

  • Counterfactual estimates were tested on random-recommender trajectories for users in the normal dataset, using smoothing estimates from their usual validation trajectories.Figure 14 reports this evaluation.Source: §I · p. 22, Figure 14 · p. 23
  • Preference prediction was similar to oracle under correct and mis-specified choice models, with slightly higher losses and accuracy very slightly higher by random variability.The initial preference estimation network results are shown in Figure 13.Source: §I · p. 22, Figure 13 · p. 23
  • The counterfactual network generally performed somewhat worse, likely because smoothing estimates are approximate and the evaluation policy differed from the training policy.Source: §I · p. 22, Figure 14 · p. 23
Loading 2204.11966v2…