Source-linked AI summary
DEAR: Deep Reinforcement Learning for Online Advertising Impression in Recommender Systems
Xiangyu Zhao, Changsheng Gu, Haoshenglun Zhang, Xiwang Yang, Xiaobing Liu, Jiliang Tang, Hui Liu
TL;DR
Most RL-based advertising methods focus on ad revenue while overlooking effects on recommendation user experience, despite the trade-off between ad frequency, suitability, and revenue. DEAR uses deep reinforcement learning to jointly choose whether to insert an ad and, if so, which ad and where. Experiments on a real-world short-video site demonstrate the framework’s effectiveness.
Problem
Most RL-based advertising work prioritizes ad revenue while ignoring potential negative effects on the user experience of recommended items, although excessive or unsuitable ads can hurt experience and too few can reduce revenue.
Source: Introduction · p. 1Method
DEAR models recommendation-list advertising as a long-run reinforcement-learning task and jointly chooses whether to insert an ad and, if so, which ad and location.
Source: Introduction · p. 2, Problem Statement · p. 2Results
Experiments on a real-world short-video site demonstrate the effectiveness of the proposed framework.
Source: Introduction · p. 2, Conclusion · p. 7Takeaways & Limitations
The reported effectiveness is supported within the evaluated short-video-site setting.
Source: Introduction · p. 2, Conclusion · p. 7The setup assumes the platform’s recall mechanism preselects candidate ads and focuses on platform income and user experience.
Source: The Optimization Task · p. 5
Abstract
from arXiv · showhide
With the recent prevalence of Reinforcement Learning (RL), there have been tremendous interests in utilizing RL for online advertising in recommendation platforms (e.g., e-commerce and news feed sites). However, most RL-based advertising algorithms focus on optimizing ads' revenue while ignoring the possible negative influence of ads on user experience of recommended items (products, articles and videos). Developing an optimal advertising algorithm in recommendations faces immense challenges because interpolating ads improperly or too frequently may decrease user experience, while interpolating fewer ads will reduce the advertising revenue. Thus, in this paper, we propose a novel advertising strategy for the rec/ads trade-off. To be specific, we develop an RL-based framework that can continuously update its advertising strategies and maximize reward in the long run. Given a recommendation list, we design a novel Deep Q-network architecture that can determine three internally related tasks jointly, i.e., (i) whether to interpolate an ad or not in the recommendation list, and if yes, (ii) the optimal ad and (iii) the optimal location to interpolate. The experimental results based on real-world data demonstrate the effectiveness of the proposed framework.
Introduction
Existing RL-based advertising methods prioritize ad income while often ignoring effects on recommendation user experience. DEAR addresses the revenue–experience trade-off by jointly handling whether to insert an ad and, if so, which ad and where.
Introduction
- Most existing RL-based advertising work focuses on ad income while ignoring ads’ negative influence on recommendation user experience.The paper contrasts this with RL strategies that update through consumer interactions to maximize long-term cumulative revenue.Source: Introduction · p. 1
- Too many or inappropriate ads can degrade user experience and engagement, while too few can reduce platform advertising revenue.The paper frames the objective as maximizing ad income while minimizing negative effects on user experience.Source: Introduction · pp. 1–2
- The three decisions are linked: ad and location choices determine rewards only when the agent elects to insert an ad.Conventional DQN architectures described in the paper cannot jointly resolve all three decisions.Source: Introduction · p. 2
- DEAR jointly determines whether to insert an ad and, if so, which ad and location, addressing three interrelated advertising decisions.The paper reports that the framework was effective in experiments on a real-world short-video site.Source: Introduction · p. 2
Problem Statement
The paper models ad insertion into recommendation lists as a sequential decision problem in which an advertising agent seeks cumulative reward. Its objective balances ad income with user experience.
Problem Statement
- The state includes the user’s recommendation and ad-browsing history, the current recommendation list, and request context.The agent interacts with users by sequentially inserting ads into recommendation lists over time.Source: Problem Statement · p. 2
- The agent’s action jointly specifies whether to insert an ad and, if so, the selected ad and its location.The paper assumes at most one ad per recommendation list, while noting that extension to multiple ads is straightforward.Source: Problem Statement · p. 2
- Each action yields an immediate reward combining ad income with the ad’s influence on user experience.The paper assumes the state transition depends on the current state and action, and uses a discount factor to weight future reward.Source: Problem Statement · p. 2
- The objective is to find a policy that maximizes cumulative user reward by increasing ad income while minimizing negative effects on user experience.Source: Problem Statement · p. 2
The Proposed Framework
DEAR uses a deep reinforcement learning framework with a novel DQN architecture to address the interdependent ad-insertion decisions and the trade-off between ad income and user experience. It is trained from offline user behavior logs and then tested online.
The DQN Architecture for Online Advertising
- DEAR conditions location Q-values on a candidate ad and adds a no-ad option, jointly choosing whether, which ad, and where with O(|A|) rather than O(|A|·L) forward propagations.The no-ad option is represented as a special location; the comparison is with the classical DQN requiring |A| · L forward evaluations.Source: The Proposed DQN Architecture · p. 3
The Reward Function
- The reward combines ad income with a user-experience signal based on whether the user continues browsing; α weights the experience term.The paper focuses on platform income and user experience, while a platform recall mechanism filters ads that may generate maximal revenue and removes ineligible ads.Source: The Reward Function · p. 4, The Optimization Task · p. 5
The Optimization Task
- Training minimizes a Q-network loss against a Bellman target from separated evaluation and target networks, with target-network parameters fixed during optimization.The target maximization considers candidate ads and locations, including the location representing no ad.Source: The Optimization Task · p. 5
Off-policy Training Task
- Off-policy training stores behavior-policy transitions and updates the model from minibatches sampled from a replay buffer.Rewards are obtained from offline logs, and the sampled transitions are used to update parameters according to Equation (6).Source: Off-policy Training Task · p. 5
Online Test Task
- During online testing, the trained policy chooses whether to insert an ad, users provide feedback, and the agent updates its state for the next step.The test procedure follows the transition-generating stage of training, replacing logged feedback with feedback from the target user.Source: Online Test Task · p. 5
Experiments
The experiments evaluate DEAR against representative baselines, examine its components, and test parameter sensitivity on a real short-video site. They report that DEAR outperforms the baselines and that the balance between ad revenue and user experience depends on α.
Experiments
- The experiments assess baseline performance, component contributions, and hyperparameter effects on a real short-video site.The study poses these as three evaluation questions.Source: Experiments · p. 5
Dataset
- The evaluation uses 1,000,000 temporally ordered Douyin sessions from March 1–30, 2019, with the first 70% for training/validation and the remaining 30% for testing.The data include normal recommended videos and ad videos, with platform-predicted features.Source: Dataset · p. 5
Implementation Details
- The implementation uses eight possible locations, including no ad, a discount factor of 0.95, and a replay buffer of 10,000.The ad-video reward is its revenue; the experience reward is 1 when the user continues browsing and 0 otherwise.Source: Implementation Details · pp. 5–6
Metrics
- Performance is measured by accumulated session rewards, with improvement over each baseline and the corresponding p-value also reported.The task is to decide whether, which, and where to insert an ad, rather than to predict ad clicks.Source: Metrics · p. 6
Baselines
- The baselines are W&D, DFM, GRU, and HDQN, with W&D serving as the platform’s behavior policy.The methods include recommender models adapted for ad interpolation and a hierarchical DQN.Source: Baselines · p. 6
Overall Performance Comparison
- In the overall online advertising comparison, DEAR outperforms representative baselines by a significant margin (p-value < 0.01); specifically, it outperforms HDQN, whose jointly trained off-policy HRL framework is described as unstable.Source: Overall Performance Comparison · p. 6
Component Study
- Component tests support reinforcement learning, sequential preference modeling, the proposed architecture, and selecting appropriate ads and slots.The authors report that state-dependent value and ad-location advantage functions improve performance, and that the selected ad and slot matter.Source: Component Study · pp. 6–7, Table 3 · p. 7
Parameter Sensitivity Analysis
- As α increases, user-experience reward improves while ad revenue decreases; in this setting, overall R peaks at α = 1.The authors conclude that platforms should select α according to business demands.Source: Parameter Sensitivity Analysis · p. 7
Related Work
Related work covers reinforcement-learning approaches to online advertising and to recommender systems. The cited work spans ad-delivery and bidding settings, as well as recommendation frameworks for user feedback, pages, and whole-chain problems.
RL-based online advertising
- RL-based online-advertising work includes guaranteed delivery and real-time bidding, with MDP approaches addressing repeated campaigns before budgets run out.The passages also describe cooperative publisher policies in unstable environments and bidding under multi-armed bandits.Source: Related Work · p. 7
RL-based recommender systems
- RL-based recommender research considers positive and negative feedback, page-wise recommendation, whole-chain recommendation, and user simulation.The passage also lists additional work in this category.Source: Related Work · p. 7
Conclusion
DEAR is a deep reinforcement learning framework for online advertising in recommender systems that jointly selects ad insertion decisions and balances ad revenue with user experience. Experiments on a short-video site report improved advertising performance.
Conclusion
- DEAR significantly improves online advertising performance in recommender systems in experiments on a short-video site.The paper evaluates the framework using extensive experiments based on the site.Source: Conclusion · p. 7
- DEAR jointly decides whether to insert an ad and, if so, which ad and location to use in the recommendation list.Source: Conclusion · p. 7
- The framework aims to maximize ad revenue while minimizing the negative influence of ads on user experience.Source: Conclusion · p. 7