PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Allocate Incentives for Incentivized Advertising via Offline Model-Based Reinforcement Learning

Zilin Zhao, Han Yang, Tianpei Yang, Fangsheng Huang, Yanfei Cui, Kan Peng, Yi Li, Yiming Zong, Hao Zhang, Yinsong Xue

arXiv:2608.28065v1cs.AI

TL;DR

Incentive allocation in incentivized advertising must trade off advance payouts against delayed ad revenue while accounting for sequential user carryover, a setting prior work does not jointly study. The paper formulates it as an MDP and develops Offline-MBRL with structured world modeling, conservative optimization, and independent CES screening. Across industrial evaluations and online A/B tests, the framework improves incentive allocation and CES rankings align with deployment evidence.

  • Problem

    Incentive allocation is a cost-sensitive sequential problem because incentives create ad exposures before downstream revenue is observed, while existing work does not jointly study this setting.

  • Method

    Offline-MBRL learns structured user-response and revenue models from logs, uses conservative policy optimization, and applies an independent CES to screen policies before launch.

  • Results

    Across industrial offline environments, CES consistently ranks model-based policies above corresponding model-free counterparts, with online A/B tests confirming the deployment ranking.

  • Takeaways & Limitations

    Modeling user feedback and ad revenue can improve incentive allocation without costly online RL exploration, while CES supports pre-launch policy selection.

  • Takeaways & Limitations

    Offline RL is constrained by historical delivery and cost-control policies, causing sparse incentive coverage and difficult counterfactual evaluation; CES is not full long-horizon OPE.

Abstract

from arXiv · show

Complete your ad view and grab a 5-cent bonus! In incentivized advertising, a platform promises users a bonus before observing downstream ad revenue, encouraging them to click and complete ads. It must balance the incentive promised in advance against the revenue realized afterward: insufficient incentives forfeit monetization opportunities, whereas excessive incentives reduce net profit. Because current incentives may also shape user expectations and future engagement, incentive allocation is a sequential decision problem with delayed revenue, cost sensitivity, and carryover effects. Existing work has not studied decision-making algorithms for this setting. Auto-bidding assumes available ad opportunities, while targeted promotion optimizes incentives outside the ad monetization pipeline. We formulate the problem as an MDP and develop an offline model-based RL framework for cost-controllable sequential incentive allocation. It learns a world model of user feedback and ad revenue, then performs conservative policy optimization. An independent counterfactual scorer evaluates each learned policy on held-out logs, enabling pre-launch selection without costly online exposure. Experiments on large-scale industrial data and online A/B tests show that the scorer provides a stable offline signal. The deployment path from causal inference to offline RL and then Offline-MBRL further validates the framework: MB-IQL improves per-user net profit by 7.96\% over TD3+BC, whereas reverting to plain IQL reduces it by 6.56\% (both \(p<0.0001\)).

1 Introduction

Incentivized advertising requires cost-sensitive sequential incentive allocation because incentives create ad opportunities before downstream revenue is known. The paper proposes Offline-MBRL with conservative optimization and independent offline policy screening.

  • Incentivized advertising pays users for completing ad views, unlike traditional ads that compete mainly through placement, relevance, and creative content.
  • Incentive allocation must balance advance incentive cost against delayed ad revenue, while accounting for missed exposures, excessive payouts, cost limits, and delivery targets.
  • Existing auto-bidding assumes available ad exposures, while targeted promotion optimizes incentives outside the advertising monetization pipeline.
  • Direct online RL risks incentive spending and user-experience fluctuations, while offline RL faces sparse actions, OOD action overestimation, counterfactual-evaluation difficulty, and complex behavior.
  • Offline-MBRL learns structured response and revenue models, performs conservative policy optimization, and uses an independent CES for pre-launch policy screening.
  • Industrial data, ablations, and online A/B tests show improved net profit with controlled incentive cost and CES rankings consistent with sequential online deployment evidence.

2 Incentivized Advertising

Incentivized advertising links promised payouts to downstream monetization through a delayed business pipeline. Its short-horizon user carryover makes incentive allocation a sequential problem requiring both immediate-payoff and future-state considerations.

  • Business pipeline and downstream payoff: The platform promises bonus amount a before downstream revenue is known; exposure triggers RTB revenue, while completed ad views trigger incentive payment.
  • Business pipeline and downstream payoff: Low incentives may fail to generate exposures, whereas high incentives may increase exposure or completion but make net payoff negative after payout.
  • Bounded-window user dynamics: Current incentives can affect subsequent participation, incentive expectations, and session trajectories within the same interaction window.
  • Bounded-window user dynamics: Figure 2 shows initially low acceptance, a brief higher-click propensity after acceptance, and gradual decline as incentive fatigue accumulates.
  • Bounded-window user dynamics: The process has a bounded number of consecutive incentive opportunities, with exposure outcomes affecting response probability and real-time state in later attempts.
  • Bounded-window user dynamics: The algorithm must evaluate current incentive payoff while accounting for local effects on subsequent user states, unlike single-step prediction or long-horizon lifecycle optimization.

3 Methodology

The method formulates incentive allocation as a bounded-window MDP and learns a structured world model from logged user and revenue feedback. It augments conservative offline policy optimization with short model rollouts and uses an independent scorer to screen candidate policies before launch.

  • Markov Modeling: Each incentive attempt is an MDP transition that records exposure, completion, payout, RTB revenue, and the next state.Episodes contain at most L_max attempts and terminate after no exposure or reaching the product-side cap.
  • Markov Modeling: The state combines real-time behavior and session features with offline user attributes, long-term activity, and historical ad-response features.The policy observes s_t and outputs the absolute incentive a_t; completion and revenue are zero when no exposure occurs.
  • World Model Design: The world model predicts exposure, conditional completion, revenue, and key mutable state fields, then reconstructs the full next state using unchanged fields and business rules.During rollout, exposure and completion are sampled first, revenue is zero without exposure, and the sampled reward follows the paper’s reward definition.
  • Policy Optimization with Mixed Data: Short world-model rollouts augment logged transitions, and policy learning trains the critic on their mixed distribution with conservative Q regularization.The conservative gap uses WM-induced candidate state-action pairs while logged pairs anchor their values; its implementation also controls pessimism with β.
  • Policy Optimization with Mixed Data: The model-based extension preserves each backbone’s value and actor objectives, adding world-model transitions and a targeted conservative term without introducing a planner or new policy architecture.The actor updates follow TD3+BC or deterministic IQL+DDPG+BC policy extraction, depending on the backbone.
  • Offline Counterfactual Evaluation Scorer: An independent CES predicts counterfactual request outcomes on held-out states and aggregates them by user to rank candidate policies before deployment.CES is reusable across policies but remains a support-aware ranking tool rather than an unbiased estimator of absolute online KPIs.

4 Experiment

Experiments evaluate cost-sensitive policies, CES reliability, world-model design, and conservative regularization across offline environments and online A/B tests. Results consistently support structured world models and locally reliable CES rankings under controlled support and cost.

  • Experimental setup: The study evaluates cost sensitivity, policy rankings, CES reliability, and world-model reliability across three logged environments, with DY-2 also used for online experiments.The environments contain approximately 200M, 400M, and 50M logged records, respectively.
  • Cost sensitivity: Increasing λ decreases both incentive cost and revenue while increasing net profit on DY-2, shifting budget toward high-ROI users.The experiments interpret λ as a cost-preference regime rather than only a global bid scale.
  • Policy comparison: Every structured-WM variant improves its matched model-free backbone in CES evaluation, adding training signal beyond value learning alone.The comparison uses the same CES, held-out traffic, and cost protocol.
  • Online validation: Online A/B tests establish the matched-cost ordering DESCN < TD3+BC ≈ IQL < MB IQL, consistent with CES, while removing WM reverses the model-based gain.The tests used 10% peak traffic and more than 10M cumulative hit users.
  • CES reliability: CES rankings are locally trustworthy but degrade under reduced-support evaluation, so policy hyperparameters must limit action drift.Context augmentation improves factual discrimination and conditional-revenue prediction, while support shift worsens AUC, ECE, and revenue error.
  • WM reliability: Observed request context improves world-model response, revenue, and next-state prediction, whereas autoregressive rollouts without it amplify error.These findings support local one-step rollouts when request context is available.
  • WM reliability: The world model jointly fits response, revenue, and nearly ten mutable-state prediction heads, making it a heavier multitask problem than CES.The WM uses an MMoE architecture to improve multitask performance, but its shared targets remain less accurate than CES targets.
  • Ablation: Conservative Q regularization benefits model-based methods but slightly degrades their model-free backbones on DY-2.The ablation reports normalized CES net p./u. means and standard deviations across three random-seed runs.

5 Related Work

Related work covers adjacent incentive, bidding, and offline-RL problems but does not address platform-side incentive allocation before downstream ad exposure and revenue.

  • Incentivized advertising and auto-bidding: Incentivized social advertising studies seed selection and budget allocation for social diffusion, rather than platform-side ad-watching incentives.
  • Incentivized advertising and auto-bidding: Auto-bidding optimizes advertiser-side bids for already-existing platform exposures and therefore models the revenue side, not exposure generation.
  • Targeted promotion and uplift-based incentive allocation: Targeted promotion allocates coupons, bonuses, discounts, or treatments under cost constraints, usually optimizing conversion, retention, or LTV outside the advertising monetization pipeline.
  • Targeted promotion and uplift-based incentive allocation: In incentivized advertising, the incentive determines whether an exposure is generated, so the reward combines user response, RTB revenue, and incentive payout with short-window carryover.
  • Offline and model-based RL: Offline RL addresses risky online exploration but must handle distribution shift, extrapolation error, and value overestimation from poorly covered actions.

6 Conclusion

The paper frames incentivized advertising as cost-sensitive sequential decision-making and proposes Offline-MBRL with structured world models and independent CES screening. Industrial offline and online results support the approach, while its scope remains short-horizon net-profit optimization with support-aware evaluation.

  • Conclusion: The paper formulates incentive allocation as a cost-sensitive sequential problem in which incentives precede downstream ad revenue.
  • Conclusion: Offline-MBRL learns structured user-response and revenue models, uses local world-model rollouts with conservative policy optimization, and screens policies using an independent CES.
  • Conclusion: Across industrial offline environments, CES ranks model-based policies above matched model-free counterparts, and online A/B tests confirm the deployment ranking.
  • Limitations: The formulation targets short-horizon interaction windows and net profit, while longer-term user outcomes are monitored online but not explicit optimization targets.
  • Limitations: CES is a support-aware user-level ranking tool based on request-level predictions, not a general estimator of full long-horizon counterfactual value.

A.1 Cost Alignment and Long-Term Metrics

Offline policy comparisons align candidate costs with factual logged costs before reporting net profit, while online tests provide the realized matched-cost check. The backtests broadly recover deployment decision signs, with common-reference results reinforcing the policy ordering.

  • Cost Alignment: Candidate selection matches each policy’s CES-estimated cost/u. to the factual cost/u. of logged actions before reporting CES net p./u.The protocol trains λ-conditioned candidates and selects using the same held-out users and requests.
  • Cost Alignment: Cost control operates through λ-conditioned policy training, comparable-cost candidate screening, and production scheduling under budget and risk constraints.The three layers separate policy preference, pre-launch guardrails, and deployment-time ceilings.
  • Backtest Validation: CES and FQE both recover the signs of four actual deployment decisions in pairwise backtests, without claiming unbiased online KPI estimation.The A/B-test results serve as the deployment reference for assessing offline difference signs.
  • Backtest Validation: On a common DY-2 reference, CES places IQL_B and IQL_D at +2.5% and +2.1%, while chaining Experiments C and D returns IQL to within about 1% of TD3+BC.These checks reduce the plausibility of a data-recency explanation and reinforce DESCN < TD3+BC ∼ IQL < MB IQL.
  • Reference Comparisons: The comparison set includes sequential and one-step variants, causal or bandit policies, and reference evaluations on common environments.These are presented as supporting reference results rather than extensions of the main policy-comparison claims.

B.3 Fixed-Budget Evaluation Overhead

Under a fixed eight-GPU budget, reusable CES checkpoint evaluation adds little wall time compared with target-policy-specific FQE. The broader experimental narrative interprets rankings alongside cost alignment, online evidence, scorer support, and world-model reliability.

  • Reusable Evaluation: CES reuses a pre-materialized test set and one dataset-specific scorer across policy checkpoints, enabling repeated batched inference and user-level aggregation.This repeated-use workflow is measured under the same fixed compute allocation.
  • Fixed-Budget Overhead: CES evaluation raises total training time from 20 to 21 hours, adding 5% overhead under the fixed GPU budget.The reported CES overhead excludes its one-time 1-hour dataset-level training cold start.
  • Fixed-Budget Overhead: FQE raises total training time from 20 to 30 hours, corresponding to 50% overhead under the same synchronous eight-GPU setup.FQE fits a separate evaluator for each checkpoint, unlike the dataset-specific CES reused across checkpoints.
  • Interpretation: The main experiments interpret offline ranking at comparable cost, against matched-cost online evidence, and within scorer and world-model support regimes.The appendix supplies detailed counterparts for offline metrics, online KPIs, CES support diagnostics, and world-model rollout diagnostics.

C.2.2 Analysis.

Online experiments evaluate policies under statistically matched incentive cost and use net profit, revenue, and long-term metrics to distinguish beneficial allocation from mere delivery changes. Offline diagnostics also constrain how enriched context is used and interpreted.

  • Online Outcomes: TD3+BC increases per-user net profit by 7.72% at statistically matched cost, with significant ROI, attempt, and exposure improvements and no long-term concern.The experiment supports full rollout under the stated business criterion.
  • Online Outcomes: IQL keeps cost and net profit flat but significantly reduces incentive attempts by 1.10% and exposures by 1.22%, so it represents reallocation rather than demonstrated improvement.The paper attributes this pattern post hoc to directing more cost toward a small set of higher-eCPM users.
  • Online Outcomes: MB IQL increases revenue by 6.80% and net profit by 7.96% with statistically unchanged cost and no significant monitored long-term metric changes.These matched-cost gains satisfy the reported rollout criterion.
  • Online Outcomes: Plain IQL decreases revenue by 4.77%, net profit by 6.56%, and ROI by 5.32% while attempts and exposures increase, indicating broader low-eCPM allocation.More exposure does not translate into more value in this reversal test, so the policy should not be rolled out.
  • Context and Diagnostics: Enriched request context is timestamp-masked and retained only offline for WM and CES, while the deployed Incentive Actor receives only the serving state.After conditioning on state, residual action correlations remain below 0.1, supporting the CES context design.
  • Context and Diagnostics: The paper does not formalize observability because predictability of future context and possible state augmentation remain unresolved.This scope boundary is described as orthogonal to structured one-step modeling and supported offline evaluation.

C.3.3 Cross-Support CES Evaluation.

Cross-support diagnostics compare CES trained on inner versus full action support, while rollout diagnostics examine world-model drift. CES is reliable near support, but aggressive policies and recursive rollouts expose degradation.

  • Context Construction: Figure 5’s timestamp masking prevents future-information leakage by retaining only context available at or before each incentivized-ad request.The masked context is used in D_off for WM and CES, not by the online actor.
  • Context Construction: After residualizing action and context on state, absolute correlations remain below 0.1, with only modest action-prediction improvement from adding context.The reported diagnostics show max correlation 0.066 and mean 0.045.
  • Cross-Support CES: BC and IQL (λ=1.5) remain inside inner DY-2in support across all six eCPM buckets, whereas IQL (λ=8.0) enters the gap but remains within full DY-2 support.The figure tracks mean actions by eCPM bucket and marks inner and full support bands.
  • Cross-Support CES: CES_in and CES_all nearly coincide for BC and IQL (λ=1.5), but for IQL (λ=8.0) CES_in overestimates cost/u. and revenue/u. and underestimates net p./u.The cost error dominates, and the diagnostic supports local CES ranking with policy OOD controlled through λ.
  • World-Model Rollouts: Autoregressive world-model prediction error compounds, with substantial deterioration already after one to two rollout steps.The rollout analysis aggregates prediction performance by rollout step and logged trajectory length.
  • World-Model Rollouts: One-step world-model predictions stay closer to logged mutable states than selected two-step autoregressive predictions in the qualitative t-SNE view.The t-SNE visualization complements quantitative rollout-drift analysis rather than serving as an OOD test.

D Experimental Setup

The experiments use three industrial logged environments and a shared offline policy-learning and counterfactual evaluation protocol. Model-based methods generate gated synthetic transitions, while CES computes expected user-level outcomes for policy ranking.

  • Industrial environments: Three industrial logged environments differ in platform source, historical exploration policy, and cost pressure.DY-1 and DY-2 use Douyin traffic, while DH uses Douyin’s Huoshan Version traffic; the two Douyin environments are treated separately.
  • Optimization: Offline policy-learning jobs use distributed GPU training, a global batch size of 1024, 1M optimization steps, and a cosine learning-rate schedule from 10^-3 to 10^-5.These settings apply unless otherwise stated in the experimental protocol.
  • Evaluation protocol: Each configuration uses three independent seeds with a fixed user-disjoint train/test split and seed-specific CES evaluation.Reported offline means and standard deviations aggregate the three seed-specific CES pipelines.
  • Baselines: Behavioral and offline-RL baselines include BC, 10% BC, DT, TD3+BC, and CQL, with specified behavior-cloning, return-conditioning, warm-up, and conservative-action procedures.TD3+BC uses a 20K-step warm-up, while DT conditions sequence modeling on a 60th-percentile return-to-go with context length 6.
  • World-model training: World-model rollouts sample exposure first, condition completion on exposure, set revenue to zero without exposure, and compute reward as r_λ = I − λya.The synthetic-transition ratio warms up from zero and increases to 0.5; rollouts use one step and are periodically refreshed.
  • CES evaluation: CES uses direct probabilities to compute expected counterfactual revenue and cost, then aggregates request predictions by user to remove Monte Carlo noise from policy ranking.The held-out dataset is reused across candidate policies and checkpoints through an inference-only evaluation path.
  • Representations: The networks combine a position-free Transformer feature encoder with an MLP prediction head, while states include numerical, bucketized, and categorical feature representations.Numerical inputs are clipped to their 1st–99th percentile range and sigmoid-normalized; actions use Yeo–Johnson transformation and rewards use min–max normalization.

E D4RL Benchmark Results

The D4RL benchmark tests model-based extensions under a standard transition-and-reward setup distinct from the industrial incentivized-advertising task. Across four paired HalfCheetah comparisons, the MB extension outperforms its plain counterpart in three cases.

  • Benchmark setup: D4RL HalfCheetah normalized returns are reported for Medium and Medium-Replay dataset types, with higher scores better.Public baseline and COMBO results come from prior reported evaluations, while MB IQL and MB TD3+BC use the authors’ implementations and test results.
  • Setting comparison: The benchmark uses long-horizon control trajectories with directly observed scalar rewards, unlike the industrial task’s gated episodes and posterior advertiser revenue.The comparison isolates generic model-based extensions from the domain-structured industrial world model.
  • Method: The benchmark replaces the domain-structured world model with a standard MLP-based world model and applies the reported hyperparameters without benchmark-specific tuning.This replacement is the principal benchmark-specific methodological change described in the supplied passage.
  • Results: Across four paired comparisons, the model-based extension outperforms its plain counterpart in three cases.The result summarizes the paired comparisons reported in Table 14 rather than a single dataset type or metric row.
Loading 2608.28065v1…