PaperlayerView exact revision on arXiv

Source-linked AI summary

FARE: Deep Reinforcement Learning For Fair Exposure Constrained Uncertainty Aware Financial Content Personalization

Arundeep Chinta, Lucas Vinh Tran, Jay Katukuri

arXiv:2609.31890v1cs.LGcs.AIcs.IR

TL;DR

Engagement-focused ranking can concentrate exposure on high-CTR content, while existing approaches do not use prediction uncertainty to guide fairness corrections. FARE applies SOV constraints downstream of a frozen CTR model, and its policies show differing performance across evaluation settings.

  • Problem

    Fairness-aware re-ranking often treats relevance predictions as deterministic, while uncertainty-aware methods use prediction uncertainty for exploration rather than to guide fairness adjustments.

    Source: §1 · p. 2

  • Method

    FARE is a plug-in execution layer that uses CTR predictions and uncertainty to adjust rankings toward SOV constraints, with a proportional controller and two learned policies.

    Source: §3.2 · p. 5, §3.2.1 · p. 7

  • Results

    On KuaiRand-Pure, FARE-PPO achieves a 16.7% reduction under DCG weighting and outperforms FARE-ES under Top-3 (25.4% vs. 12.7%); at K=10, FARE-PPO achieves a 0.8% reduction.

    Source: §4.3.3 · p. 11, §4.4.1 · p. 11

  • Takeaways & Limitations

    Because FARE operates downstream of the predictor, the CTR model can be improved or replaced without retraining the fairness layer.

    Source: §6 · p. 13

    The evaluation shows every candidate in the slate and does not address filtering a large catalogue to a short slate.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Content personalization systems in financial services must ensure fair exposure across diverse offerings-a requirement driven by contractual obligations and the need to prevent "rich-get-richer" dynamics where content with high click-through rate (CTR) dominates while other relevant products receive minimal visibility. Share of Voice (SOV) constraints, which guarantee each content category a target fraction of top-position exposure, address this by promoting product diversity and balanced user discovery. While re-ranking layers atop CTR models are common in practice, we propose two key novelties: (1) framing SOV-constrained ranking as a deep reinforcement learning problem analogous to constrained trade execution in algorithmic finance, and (2) explicitly incorporating CTR prediction uncertainty into the agent's state space and policy design-enabling larger ranking adjustments for high-uncertainty predictions where deviation from CTR-optimal ordering is less costly. We introduce FARE (Fair Ranking Executor), a modular uncertainty-aware execution layer that translates any black-box CTR model's predictions into SOV-fair rankings without retraining the underlying model. Our uncertainty-weighted proportional control policy (FARE-PC) and learned neural policies (FARE-ES, FARE-PPO) demonstrate that uncertainty-aware approaches can substantially reduce SOV deviation from fairness targets while minimizing engagement loss, with gradient-free evolution strategies outperforming policy gradient methods on synthetic data and the ordering reversing on KuaiRand-Pure.

1 INTRODUCTION

Engagement-only ranking can reinforce high-CTR content’s exposure, while fairness-aware methods often ignore prediction uncertainty. FARE adapts signal/execution separation to apply SOV constraints downstream of the CTR model.

1 INTRODUCTION

  • High-CTR content can dominate top positions and accumulate the impressions and clicks that reinforce its advantage, limiting visibility for other relevant offerings.The section motivates balanced exposure through user interest in diverse content and operators’ contractual requirements.Source: §1 · pp. 1–2
  • Fairness-aware re-ranking often treats relevance predictions as deterministic, while uncertainty-aware methods use uncertainty for exploration rather than fairness adjustments.FARE uses prediction uncertainty to prioritize adjustments where the model is least confident and deviation from CTR-optimal ordering is least costly.Source: §1 · p. 2

1.1 DESIGN INSPIRATION FROM ALGORITHMIC TRADING

  • FARE adapts algorithmic trading’s separation of signal generation from constrained execution, with the CTR model supplying predictions and FARE acting as the execution layer.The trading analogy is a separate execution agent translating signals into orders that meet operational constraints such as VWAP targets.Source: §1.1 · p. 2, Figure 1 · p. 2
  • FARE is a downstream plug-in that adjusts rankings to satisfy SOV constraints without retraining the CTR model, and allows targets to be changed independently.The framework can work with uncertainty-aware CTR models such as Thompson Sampling, deep ensembles, or Bayesian methods.Source: §1.1 · p. 2

1.2 CONTRIBUTIONS

  • FARE frames SOV-constrained ranking as deep reinforcement learning and uses prediction uncertainty to scale fairness corrections where deviation from CTR-optimal ordering is cheapest.Its three execution policies are FARE-PC, FARE-ES, and FARE-PPO; the learned policies’ relative performance varies across regimes.Source: §1.2 · p. 3

2 PROBLEM FORMULATION

The formulation treats a frozen CTR model’s predictions as inputs to a ranking policy that balances top-position SOV targets against CTR loss. The evaluation considers slates in which every candidate is shown.

2.1 SETTING

  • The CTR model is treated as a black box that provides each tile’s mean prediction and predictive variance for a user.These mean and variance predictions are the inputs to the downstream ranking process.Source: §2.1 · p. 3
  • The CTR-Only baseline samples scores from the prediction distributions and ranks tiles by descending sampled score; FARE adjusts means downstream before sampling.The deterministic alternative ranks by descending mean prediction, ignoring both uncertainty and fairness.Source: §2.1 · p. 3
  • FARE’s evaluated setting fixes the candidate set to the slate, with every candidate shown and at most 10 exposure groups; it does not address filtering a large catalogue to a short slate.The controller is agnostic to allocating exposure over items or groups, but the evaluated setting excludes candidates receiving no exposure because they were filtered out.Source: §2.1 · p. 3

2.2 SHARE OF VOICE CONSTRAINT

  • Share of Voice is a tile’s fraction of top-position appearances across users and days.The definition counts whether a tile appears in position 1 for each user-day.Source: §2.2 · p. 3
  • Targets may differ by tile; the default equal-allocation target is τi = 1/K, and FARE can handle arbitrary targets.Targets may reflect external contracts or business objectives; merit-proportional schemes are not evaluated, and dynamic targets are left for future work.Source: §2.2 · pp. 3–4

2.3 CONSTRAINED OPTIMIZATION PROBLEM

  • The constrained-ranking objective is analogous to trade execution: meet SOV targets within tolerance while minimizing CTR loss.The policy translates CTR signals into rankings, just as a trade executor translates price signals into orders satisfying VWAP constraints while minimizing execution cost.Source: §2.3 · p. 4

3 METHODOLOGY

FARE takes mean and uncertainty predictions from synthetic or learned CTR models, then applies a separate execution layer to meet Share of Voice targets. Its three policies use uncertainty-aware proportional control or learned strategies, with training based on synthetic or replayed data.

3.1 INPUT GENERATION

  • FARE accepts mean and variance predictions from either synthetic generation or a learned Gaussian MLP, and changes means while preserving variances for score sampling.Its learning-based policies receive CTR predictions, SOV, deficits, and time as state; the reward balances SOV deviation against position-weighted CTR loss.Source: §3.1 · p. 4, §3.2 · p. 6

3.1.1 APPROACH 1: SYNTHETIC CTR ESTIMATE GENERATION

  • Synthetic inputs impose a decreasing tile-mean ordering while varying uncertainty independently, and the generator supplies both evaluation inputs and training environments.The resulting baseline places Tile 1 at about 28% of Position 1 exposure and Tile 5 at about 11.5%.Source: §3.1.1 · p. 4

3.1.2 APPROACH 2: GAUSSIAN MLP FOR UNCERTAINTY-AWARE CTR ESTIMATION

  • For KuaiRand-Pure, a multi-head Gaussian MLP predicts CTR means and variances from user features, and its predictions feed FARE and replay-based policy training.Predicted tile ordering is non-monotonic, baseline Position 1 SOV ranges from 23.3% to 15.1%, and median predicted variance is about 0.03.Source: §3.1.2 · pp. 4–5

3.2 FARE ARCHITECTURE

  • FARE freezes the upstream CTR model and adjusts its outputs downstream through proportional-control, evolution-strategy, or PPO policies.The SOV target does not enter CTR-model training; the learning-based policies use a replay simulator, while FARE-PC is reactive.Source: §3.2 · pp. 5–7

3.2.1 FARE-PC: UNCERTAINTY-WEIGHTED PROPORTIONAL CONTROL

  • FARE-PC adjusts each tile’s mean in proportion to its SOV deficit, uncertainty, and inverse remaining horizon, boosting underexposed tiles and penalizing overexposed ones.The controller is self-correcting by construction, but convergence is not guaranteed; residual error grows with larger K and under DCG weighting.Source: §3.2.1 · p. 7

3.2.2 FARE-ES: EVOLUTION STRATEGY POLICY

  • FARE-ES trains a neural policy using population-based evolution-strategy search, updating weights toward perturbations with better trajectory fitness.Its updates do not use backpropagation.Source: §3.2.2 · p. 8

3.2.3 FARE-PPO: PROXIMAL POLICY OPTIMIZATION

  • FARE-PPO uses a combined actor-critic network and clipped PPO updates to learn a policy under the same MDP formulation.The implementation uses a shared two-layer MLP with actor and critic heads.Source: §3.2.3 · p. 8

3.3 COMPUTATIONAL COMPLEXITY

  • FARE-PC has O(K) inference cost, no stored model weights, and one tuned hyperparameter; FARE-ES and FARE-PPO each require an MLP pass per impression.ES evaluates full trajectories for each generation update, whereas PPO performs several gradient epochs per trajectory.Source: §3.3 · p. 8

4 EXPERIMENTS

The experiments compare FARE with reference policies on synthetic and KuaiRand-Pure data across target types, exposure weightings, tile counts, and a joint diversity objective. They assess fairness, model-estimated engagement loss, and ranking stability, with results varying by setting and policy.

4 EXPERIMENTS

  • The comparison uses CTR-Only, Max-Deficit, and Quota-CTR references and evaluates SOV Error, PWCL, and ranking stability; PWCL estimates utility loss because counterfactual clicks are unavailable.Tests include synthetic and KuaiRand-Pure settings, uniform and non-uniform targets, multiple SOV weightings, and increasing tile counts; synthetic learned-policy training and testing use separate days.Source: §4 · p. 8, §4.1 · p. 8, §4.3 · p. 9

4.3.1 UNIFORM SOV TARGETS

  • On uniform targets, FARE-PC reduced SOV error by 90.3% at 2.64% PWCL on synthetic data and by 91.8% (0.122 → 0.010) on KuaiRand-Pure.The reported comparisons appear in Table 1 and Table 3, respectively.Source: §4.3.1 · pp. 9–10, Table 1 · p. 9, Table 3 · p. 10
  • On the test set, the reported final SOV distribution uses one representative seed; FARE-PC dominates the Pareto frontier, achieving the best SOV correction at the lowest engagement cost.FARE-PC’s final SOV Error is 0.024, versus 0.034 for FARE-ES. FARE-PPO ends with an approximately 9 percentage point spread across tiles, indicating incomplete optimization.Source: Table 2 · p. 10, §4.3.1 · p. 10

4.3.2 NON-UNIFORM SOV TARGETS

  • For asymmetric synthetic targets, FARE-PC and FARE-ES each reduce SOV error by about 78%, while FARE-PPO achieves 32.9%; on KuaiRand-Pure, PC and ES achieve 83.0% and 81.6%.The synthetic baseline SOV Error is 0.547, compared with 0.247 under uniform targets.Source: §4.3.2 · p. 10

4.3.3 MULTI-POSITION SOV WEIGHTING

  • Across Top-1, Top-3, and DCG weighting on synthetic data, FARE-PC improves fairness in all three schemes, while ES and PPO degrade it under DCG; on KuaiRand-Pure, all three methods improve under DCG.Under synthetic Top-3, FARE-ES achieves a 73.7% reduction versus 70.8% for FARE-PC; on KuaiRand-Pure, PPO achieves 16.7% under DCG and leads ES under Top-3, 25.4% to 12.7%.Source: §4.3.3 · pp. 10–11

4.3.4 MULTI-OBJECTIVE: SOV + CATEGORY DIVERSITY

  • With a diversity objective on synthetic data, FARE-PC's SOV Error rises 26× as λdiv increases from 0 to 5, whereas FARE-ES rises 2.6×; the KuaiRand-Pure diversity objective is constant and not swept.At λdiv=2.0, FARE-ES has lower SOV Error than FARE-PC, 0.063 versus 0.175; the authors recommend PC when fairness dominates and ES when λdiv ≥2 in this setting.Source: §4.3.4 · p. 11

4.4.1 EFFECT OF INCREASING THE NUMBER OF TILES

  • As tile count rises from 3 to 10, FARE-PC degrades gracefully from 94.4% to 77.1% SOV-error reduction, while FARE-PPO reaches 0.8% reduction at 10 tiles.FARE-PC's PWCL remains between 2.45% and 2.87%; the authors attribute its graceful degradation to computing each tile's adjustment independently.Source: §4.4.1 · p. 11

4.4.2 SENSITIVITY ANALYSIS: CONTROL GAIN α; 4.4.3 ROBUSTNESS TO UNCERTAINTY MISCALIBRATION

  • For FARE-PC, increasing α from 0.5 to 2.0 lowers uniform-target SOV Error from 0.125 to 0.024 as PWCL rises from 1.4% to 2.6%; fairness improvement versus CTR-Only persists across a 16× uncertainty-scale range.The authors describe α as tunable to an operator's tolerance for CTR degradation and caution that larger uncertainty scales act like raising α, with an analogous PWCL cost.Source: §4.4.2 · pp. 11–12, §4.4.3 · p. 12

5 RELATED WORK

Prior work spans exposure-constrained re-ranking, uncertainty-driven exploration, and online allocation. FARE is distinguished by using upstream prediction uncertainty to scale a closed-loop correction toward an exogenous, two-sided exposure target.

Deep RL for Trade Execution; Uncertainty in Reinforcement Learning

  • Deep RL has been used for trade execution, while much RL uncertainty work concerns the RL system itself; FARE instead uses an external predictor's σ² as an execution-state feature [Osband et al., 2016; Bellemare et al., 2017; Chua et al., 2018].The paper adapts the signal-forecasting/execution separation described by Nagy et al. (2023).Source: §5 · p. 12

Uncertainty-Aware Ranking

  • Recommendation methods including Thompson Sampling, UCB, ensembles, and predictive-uncertainty exploration use uncertainty for exploration, unlike FARE's use of it as an execution signal [Lakshminarayanan et al., 2017].The authors distinguish this from both exploration and a post-hoc fairness constraint.Source: §5 · p. 12

Fair Exposure and Re-Ranking in Recommendation and Retrieval

Budget-Constrained Online Allocation

  • Budget-constrained allocation methods track consumption against a target over a known horizon, but use hard budget ceilings rather than signed corrections for exposure targets [Shamsi et al., 2014; Balseiro et al., 2021].Unlike those methods, FARE scales its correction by upstream predictive uncertainty.Source: §5 · pp. 12–13

Control-Theoretic Enforcement and Uncertainty for Fairness

  • FARE-PC shares proportional-control mechanisms with FairCo and uncertainty-bounded movement with PUFR, but uses upstream uncertainty to scale a closed-loop correction over cumulative exposure deficit [Morik et al., 2020; Heuss et al., 2023].The authors identify the uncertainty gain as their contribution and note that empirical comparison against both methods remains outstanding.Source: §5 · p. 13

Evaluation of Fairness in Recommendation

  • The evaluation does not use ContentWise Impressions or EB-NeRD and reports constraint-violation metrics rather than consumer- and provider-side fairness metrics [Pérez Maurera et al., 2020; Kruse et al., 2024; Malitesta et al., 2025; Diaz et al., 2020].Only ContentWise retains display order needed for position-weighted exposure, according to the passage.Source: §5 · p. 13

6 CONCLUSION

FARE applies Share-of-Voice constraints downstream of a frozen CTR model, using an execution-layer design adapted from algorithmic trading. Its modularity separates prediction from exposure allocation, and its policies differ in how they compute mean adjustments.

6 CONCLUSION

  • FARE enforces Share-of-Voice constraints on a frozen CTR model's output, separating CTR forecasting from exposure-budget execution over a horizon.The layer can be used without retraining the CTR model and is agnostic to whether allocation units are items or groups.Source: §6 · p. 13
  • The execution layer can remain in place when the CTR model is improved or replaced because it does not enter the predictor's training objective.The paper compares this separation to a VWAP executor working against a volume schedule.Source: §6 · p. 13
  • FARE-PC computes mean adjustments with an uncertainty-scaled control law, whereas FARE-ES and FARE-PPO use learned policies; policy choice depends on whether one or several objectives must be balanced.Source: §6 · p. 13

7 DISCLAIMER

The appendix provides a disclaimer and algorithm details for the CTR baseline, synthetic generator, and FARE policies. The algorithms adjust predicted means before sampling scores to produce rankings.

7 DISCLAIMER

  • JPMorganChase disclaims warranties and liability regarding the information and states that the document is not investment research, advice, or a recommendation or solicitation.Source: §7 · p. 13

A ALGORITHM DETAILS

  • CTR-Only samples each tile's score from its predicted Gaussian and ranks by sampled score; FARE changes this procedure by adjusting the predicted means before sampling.Source: A ALGORITHM DETAILS · p. 16
  • The synthetic generator sets decreasing base means to produce the desired tile ordering, adds day, user, and interaction variation, then clips the resulting means and variances.Source: A ALGORITHM DETAILS · p. 16
  • FARE-PC computes the current exposure deficit against the target and scales its adjustment by uncertainty and time remaining before adjusting the mean.Source: A ALGORITHM DETAILS · p. 16
  • FARE-ES perturbs policy-network weights across a population, evaluates each policy on an episode, and updates the weights from fitness.Source: A ALGORITHM DETAILS · p. 17
  • FARE-PPO uses an actor-critic policy on a state containing predictions and exposure information, applies mean adjustments, and updates the policy using a clipped objective.Source: A ALGORITHM DETAILS · p. 17

B.1 SYNTHETIC GENERATOR CONSTANTS

The synthetic generator uses ordered base CTR means with independently varying base variances, then adds day-, user-, and interaction-level noise. Figure A1 depicts the resulting position and user-level distributions.

B.1 SYNTHETIC GENERATOR CONSTANTS

  • Base means [0.75, 0.65, 0.55, 0.45, 0.35] decrease by tile, while base variances [0.35, 0.40, 0.50, 0.45, 0.40] vary independently.Source: §B.1 · p. 17
  • Day, user, and user–day interaction noise use scales 0.03, 0.08, and 0.02; final means and variances are clipped to [0, 1] and [0.2, 0.8].Source: §B.1 · p. 17
  • Figure A1 shows position heatmaps, the Position 1 SOV distribution, base tile means and variances, and user-level variation across 10,000 users.Source: Figure A1 · p. 18

B.2 KUAIRAND-PURE TILE CONSTRUCTION

KuaiRand-Pure categories are grouped into five tiles, and the predictor and exposure constraint are defined at the category level.

B.2 KUAIRAND-PURE TILE CONSTRUCTION

  • The five tiles are the most frequent Level-1 categories; videos outside them are dropped rather than grouped into an “other” tile.Each video is assigned to one category using the first Level-1 tag in its metadata.Source: §B.2 · p. 18
  • Daily CTR is clicks divided by impressions for each user, day, and category; entries without impressions are masked from training.The category aggregation matches the group-level exposure constraint.Source: §B.2 · p. 18

C ADDITIONAL EXPERIMENTAL RESULTS

The additional experiments cover uniform and non-uniform targets, multi-position weighting, diversity trade-offs, tile-count scaling, control-gain sensitivity, and uncertainty miscalibration.

C.1 UNIFORM SOV TARGETS

  • Table A1 reports KuaiRand-Pure test-set SOV distributions for five tiles against a 20% target per tile.The table gives one representative seed.Source: Table A1 · p. 19

C.2 NON-UNIFORM SOV TARGETS

  • Table A2 reports synthetic test metrics for non-uniform targets of 10%, 15%, 20%, 25%, and 30%.Table A3 gives the corresponding final SOV distribution for one representative seed.Source: Table A2 · p. 19, Table A3 · p. 19

C.3 MULTI-POSITION SOV WEIGHTING

  • Tables A4 and A5 report multi-position SOV performance across weighting schemes for uniform 20% targets.Table A4 covers the general evaluation and Table A5 covers KuaiRand-Pure.Source: Table A4 · p. 19, Table A5 · p. 20

C.4 MULTI-OBJECTIVE: SOV AND DIVERSITY

  • Table A6 compares SOV Error, PWCL, and Category Diversity across methods in a multi-objective λdiv sweep.Its CTR-Only baseline has SOV Error 0.247 and diversity 2.300.Source: Table A6 · p. 20

C.5 EFFECT OF INCREASING THE NUMBER OF TILES

  • Table A7 reports performance as the number of tiles K increases under uniform targets τ = 1/K.Values are reported as mean ± standard deviation over seeds.Source: Table A7 · p. 20

C.6 CONTROL-GAIN SENSITIVITY

  • Under uniform 20% targets, Figure A3 shows SOV Error decreasing monotonically with α while PWCL cost increases with diminishing marginal cost.Per-tile SOV converges to target; the CTR-Only baseline error is 0.247.Source: Figure A3 · p. 20

C.7 UNCERTAINTY MISCALIBRATION

  • Figure A4 plots FARE-PC SOV Error relative to CTR-Only across uncertainty scale factors from 0.25× to 4×.The caption specifies the comparison and tested scale range, but not the outcome.Source: Figure A4 · p. 21

D LIMITATIONS

The study’s evidence is limited by its comparisons, metrics, offline evaluation, and variability in learned-policy results.

D LIMITATIONS

  • The baselines are implemented reference policies, not published fair re-ranking systems, and the authors identify FairCo as the most significant omitted direct comparison.The authors therefore frame their results as a fairness–engagement frontier relative to naive enforcement, not evidence of state-of-the-art superiority.Source: D LIMITATIONS · p. 21
  • SOV Error and PWCL are setting-specific metrics, and the paper reports neither standard accuracy metrics nor community fairness metrics.This limits comparison with published results.Source: D LIMITATIONS · p. 21
  • PWCL is a noisy, model-based estimate of utility loss against the sampled CTR-Only ordering, not measured engagement loss.The policy and CTR-Only scores are sampled independently, and no counterfactual click data exists for FARE’s rankings.Source: D LIMITATIONS · p. 21
  • FARE-PPO trails FARE-ES on synthetic data but leads it on KuaiRand-Pure, with high variance across seeds in both settings.The authors attribute instability to weak per-step credit assignment but do not rule out tuning.Source: D LIMITATIONS · pp. 21–22
  • Engagement loss is estimated from predicted CTR rather than observed clicks, and aggregate PWCL does not show how costs vary across users.The study is offline, does not assess consumer-side fairness, and averages three seeds without significance testing.Source: D LIMITATIONS · p. 22

E DISCUSSION

The discussion bounds transfer to small fully exposed slates, one aggregated dataset and one predictive model, and chosen experimental settings; future work proposes broader evaluations and extensions.

E DISCUSSION

  • Experiments use at most 10 groups with every candidate shown, so results concern ordering rather than selection from a large catalogue.They do not establish transfer to retrieval settings where long-tail coverage and which items are surfaced constrain fairness.Source: E DISCUSSION · p. 22
  • The real-data evaluation uses one dataset aggregated to its top five categories, dropping videos outside them and losing item-level and long-tail exposure information.Re-ranking exposure is simulated from a frozen predictor; performance inherits its errors, and engagement loss is not measured with counterfactual clicks.Source: E DISCUSSION · p. 22
  • FARE is evaluated with one multi-head Gaussian MLP, without testing alternative uncertainty-aware predictors or assessing their variance calibration.The synthetic rescaling study does not compare genuinely differently calibrated models.Source: E DISCUSSION · p. 22
  • The non-uniform target vector is an adversarial synthetic stress test, and diversity weights are swept over an arbitrary grid rather than representative operating points.The discussion calls for targets based on real mandates and diversity-weight ranges based on operator tolerance.Source: E DISCUSSION · p. 22, F FUTURE WORK · p. 23

F FUTURE WORK

  • Future work prioritizes common-protocol comparisons with applicable fixed-slate methods and community-standard metrics.It also proposes measuring per-user PWCL and observed clicks, and extending the task to candidate selection in larger catalogues.Source: F FUTURE WORK · pp. 22–23
  • Proposed extensions include joint tile optimization, offline RL, time-varying targets, and integral or derivative control terms.The authors also propose testing alternative CTR models with calibration diagnostics and evaluating exposure on impression-logged datasets.Source: F FUTURE WORK · p. 23

Related papers

Loading 2609.31890v1…