PaperlayerView exact revision on arXiv

Source-linked AI summary

DMRL: Document-Mediated Reinforcement Learning for Skill Optimization in Advertising Recommendation

Wei Zhang, Hongji Li, Song Sun, Peng Yu, Xue Yang, Lei Zhao, Peng Jiang

arXiv:2609.02170v1cs.LG

TL;DR

Advertising recommendation requires tuning parameters under costly, delayed, and heterogeneous feedback, while existing skill optimization lacks principled reward attribution to document edits. DMRL uses bilevel structured document editing with DRPO and LRP to address edit-level credit assignment and long-term prediction, and reports significant improvements over baselines in large-scale deployment.

  • Problem

    Existing advertising tuning is labor-intensive, and skill-guided methods lack a principled mechanism for attributing observed rewards to specific document edits under delayed, heterogeneous outcomes.

  • Method

    DMRL uses an upper-level agent for structured skill-document edits, a frozen lower-level task agent for A/B-tested interventions, DRPO for robust advantage estimation, and LRP for heterogeneous long-term reward prediction.

  • Results

    DMRL was deployed on a large-scale advertising platform and demonstrated significant improvements over baselines across key advertising metrics.

  • Takeaways & Limitations

    DMRL provides a deployable framework for optimizing skill edits toward long-term advertising outcomes in a real-world industrial environment.

  • Takeaways & Limitations

    Conclusions are restricted to within-table comparisons, because experiments across tables use non-overlapping windows, traffic allocations, and conditions.

Abstract

from arXiv · show

Advertising recommendation requires continuously tuning complex system parameters while balancing commercial returns and user experience. Recent work has introduced large language models (LLMs) with skill documents to assist this labor-intensive process, but skill optimization remains largely prompt-driven, lacking a principled mechanism to attribute rewards to specific document edits. To address this limitation, we propose Document-Mediated Reinforcement Learning (DMRL), a skill self-evolution framework that models skill document optimization as a sequence of structured editing actions. In DMRL, an upper-level agent performs controlled document edits, while a frozen lower-level task agent evaluates their effects through A/B testing. To address credit assignment and long-term outcomes, we introduce two key components: (1) Dual-Relative Policy Optimization (DRPO), a post-training policy optimization method for robust and risk-aware advantage estimation; and (2) Long-term Reward Predictor (LRP), which estimates long-term outcomes by modeling population heterogeneity with disentangled representation learning and cross-attention transfer. DMRL was deployed on a large-scale short-video ads platform and extensive empirical evaluation shows that DMRL outperforms state-of-the-art baselines across key advertising metrics

1 Introduction

Advertising recommendation must balance commercial returns with user experience, but existing tuning and skill-guided approaches remain costly, expert-dependent, and weak at attributing rewards to document edits. DMRL addresses these gaps by decomposing skill optimization into structured editing, long-term outcome estimation, and edit-level reward attribution.

  • Motivation: Manual tuning requires substantial human effort and computational resources, while HPO remains dependent on domain expertise and iterative experimentation.The process also scales effort with parameter count, is difficult to transfer across practitioners, and lacks systematic exploration.
  • Motivation: LLM-based skill documents externalize operational knowledge into reusable procedures, decision rules, and tool-use patterns for advertising optimization.They provide an executable and reusable interface for encoding tuning knowledge.
  • Motivation: Existing skill-guided methods primarily refine text or prompts without principled attribution of observed rewards to specific document edits.This limits traceable optimization of structured skill documents.
  • Motivation: User segments differ substantially in short-to-long outcome mappings and temporal dynamics, complicating long-term reward prediction from aggregated data.Figure 1 measures short-term signals within 1 day and long-term outcomes within 7 days.
  • DMRL: DMRL decomposes skill optimization into long-term outcome estimation and edit-level reward attribution through a structured skill-optimization framework.Its upper-level optimizer modifies documents, while the lower-level task agent performs parameter interventions.
  • DMRL: DRPO provides robust, risk-aware advantage estimation, while LRP predicts long-term rewards using disentangled representations and historical transfer.The framework was deployed on a real-world advertising platform and demonstrated significant online improvements.

2 Related Work

Related work spans reusable LLM-agent skills, iterative skill refinement, reinforcement-learning post-training, and delayed-feedback modeling. These approaches motivate DMRL but leave systematic local edit attribution and long-term heterogeneous feedback insufficiently addressed.

  • Agent skills: ReAct, Toolformer, and later agent-skill work show that explicit procedures and external knowledge can improve LLM-agent task performance.Reusable skills systematize externalized operational knowledge for agent behavior.
  • Skill evolution: EvoSkill and SkillForge iteratively improve skills through failure analysis, but mainly use prompt-level refinement or whole-skill regeneration.Their local document modifications and causal attribution remain limited.
  • RL post-training: PPO, DPO, RLVR, and GRPO variants address policy optimization, reward modeling, or normalization issues in LLM post-training.The cited variants include DAPO, Dr. GRPO, and GSPO.
  • Delayed feedback: Delayed-feedback research addresses immature labels and lagged conversions through correction, adaptation, elapsed-time modeling, and related methods.These methods target delayed outcomes in advertising and recommendation settings.

3 Method

DMRL is a bilevel closed-loop framework in which an upper-level policy edits structured skill documents and a frozen lower-level task agent uses them to intervene on advertising parameters. DRPO and a two-stage training strategy support reliable attribution under delayed feedback.

  • Framework: DMRL uses an upper-level policy to iteratively edit structured skill documents and a frozen lower-level task agent to adjust advertising A/B parameters.The skill document acts as the semantic interface between the two levels.
  • Framework: The framework separates skill optimization from parameter intervention, allowing document edits to mediate downstream system changes.Figure 3 presents this integration of upper-level skill optimization and lower-level parameter intervention.
  • Training: A two-stage strategy first optimizes LRP for long-term reward estimation and then optimizes DRPO for policy learning.This ordering is intended to stabilize reward estimation and advantage computation.

3.1 Long-term Reward Predictor

LRP predicts delayed long-term rewards by combining population-agnostic dynamics, population-specific deviations, and transferable historical response patterns. Disentangled encoders, a continuously updated memory bank, cross-attention retrieval, and instance-wise gating produce the final reward estimate.

  • Representation: LRP estimates long-term rewards from short-term signals using disentangled representation learning and cross-attention-based historical transfer.The design targets delayed outcomes and heterogeneous short-to-long mappings.
  • Representation: The population-agnostic encoder models universal short-to-long dynamics, while the population-specific encoder captures group-dependent deviations.The first excludes population features; the second incorporates them.
  • Historical transfer: A continuously updated memory bank supplies historical experiments for similarity-based retrieval through scaled dot-product attention.Retrieval matches short-term signals, user populations, and intervention settings while aggregating transferable response patterns.
  • Historical transfer: Historical experiments are encoded into key-value pairs, with keys representing coarse conditions and values storing fine-grained early response dynamics.The value includes within-day fluctuation trajectories informative for long-term reward.
  • Reward prediction: LRP fuses the disentangled and retrieved representations with instance-wise gating before decoding the predicted long-term reward.The gate reduces reliance on historical patterns when they are less relevant to the current instance.

3.2 Dual-Relative Policy Optimization

DRPO addresses unstable and incomplete reward signals for skill-document edits by combining robust within-group normalization, treatment-control comparisons, and edit-cost penalties. It represents edits structurally and favors reward gains achieved with smaller disruption.

  • Motivation: DRPO addresses volatile rewards, unused treatment-control structure, and omitted skill-editing costs in standard GRPO.These limitations can destabilize advantage estimation and fail to account for the disruption caused by document changes.
  • Edit cost: Edit cost combines structure-dependent risk, token-level modification magnitude, and semantic shift between the original and edited text.The components respectively capture section/action risk, surface-form change, and semantic deviation.
  • Edit cost: Smaller structural or semantic disruption is preferred when edits achieve similar reward gains, discouraging overly aggressive skill modifications.This regularizes the optimization signal toward controlled skill evolution.
  • Structured edit evaluation: Each candidate edit is represented by location, action, and content, then evaluated through an online A/B experiment against a concurrent control group.LRP produces long-term rewards for rollout and control outcomes, while each edit receives an edit-cost term.
  • Dual-relative advantage: The advantage estimate combines robust within-group comparison, treatment-control improvement, and an explicit edit-cost penalty.Median and median absolute deviation reduce sensitivity to outlier rollouts; the treatment-control term uses concurrent control outcomes as a baseline.

3.3 Two-Stage Training Strategy

The training strategy first teaches LRP to estimate delayed rewards under population heterogeneity, then freezes it while DRPO optimizes the skill-editing policy. Adversarial regularization reduces population-identifying information in the population-agnostic representation, and clipped updates constrain policy drift.

  • Two-stage training: Training proceeds in two stages: LRP is optimized for long-term reward estimation, then frozen while DRPO optimizes the skill-editing policy.This ordering is intended to stabilize reward prediction and advantage estimation.
  • Stage I: Long-term Reward Predictor: LRP predicts long-term rewards from short-term signals, population features, and intervention information while adversarial regularization encourages population-invariant dynamics.A discriminator predicts population labels from the population-agnostic representation, while the encoder is trained to impede that inference.
  • Stage I: Long-term Reward Predictor: LRP training uses a three-phase schedule: reward-prediction warm-up, discriminator pretraining with the encoder frozen, and joint adversarial optimization through a gradient reversal layer.The schedule first learns reward-informative representations, then trains population classification before reversing its gradient during joint training.
  • Stage I: Long-term Reward Predictor: During joint training, minimizing adversarial loss improves discriminator classification while gradient reversal reduces population-identifying information in the population-agnostic representation.The objective combines reward prediction error with adversarial regularization controlled by λ_adv.
  • Stage II: DRPO policy learning: After training, LRP parameters are frozen for reward estimation, and the policy uses clipped updates with KL regularization to limit excessive drift from the previous iteration.The clipped surrogate objective is applied to serialized edit tokens and their prefixes.

5 Experiment

The experiments evaluate DMRL’s effectiveness, component contributions, and implementation sensitivity in advertising recommendation. Results compare DMRL with skill-optimization and long-term reward-modeling baselines using engagement, monetization, and economic-value metrics across short- and long-term horizons.

  • Implementation Details: The evaluation partitions users into four fixed groups based on historical short-term and long-term metrics to model population heterogeneity.The groups are Low-ST Low-LT, High-ST Low-LT, Low-ST High-LT, and High-ST High-LT.
  • Evaluation Metrics: The study measures AUD, PES, and LTV at short-term and long-term horizons to capture engagement, monetization, and overall economic value.Short-term metrics include AUD@1d and PES@1d; long-term metrics include AUD@7d, PES@7d, and LTV.
  • Main Result: RQ1: DMRL achieves the highest LTV improvement of +0.052% and shows substantial, consistent improvement over other methods.The comparison includes skill-optimization and long-term reward-modeling baselines.
  • Main Result: RQ1: DRPO outperforms SAGE, SkillOpt, and SKILLRL by +0.101%, +0.028%, and +0.048%, respectively.The result compares DRPO with skill-optimization baselines while isolating module contributions through fixed-component comparisons.
  • Main Result: RQ1: DRPO provides a more balanced trade-off, with PES@1d +0.644%, AUD@7d +0.082%, and PES@7d +0.960% despite AUD@1d −0.020%.The comparison highlights different engagement and monetization behavior across temporal horizons.
  • Main Result: RQ1: LRP outperforms TFT and DelayAdapter in LTV by +0.037% and +0.047%, respectively, while balancing monetization, engagement, and delayed economic value.LRP uses population-specific preferences through user-group feature extraction and historical pattern transfer.

5.3 More Result: RQ2

Ablations show that LRP components complement one another, DRPO’s components improve long-term balance, and two-stage training outperforms direct optimization on monetization metrics. Backbone size and rollout count have non-monotonic effects, while LRP predictions preserve ground-truth ordering across populations.

  • LRP ablation: The full LRP model achieves the best overall LTV improvement and strongest gains on PES@1d, AUD@7d, and PES@7d.Unified representations alone are insufficient for heterogeneous short-to-long reward mappings, while memory-based transfer helps calibrate delayed reward estimates.
  • DRPO ablation: DRPO achieves the highest LTV of +0.021%, with positive gains on PES@1d (+1.197%), AUD@7d (+0.131%), and PES@7d (+0.059%).Removing normalization, control-group-relative rewards, or edit cost weakens the balance between immediate and delayed outcomes.
  • Training strategy: The two-stage strategy improves AUD@7d by +0.082%, PES@1d by +0.644%, and PES@7d by +0.96%, whereas single-stage optimization produces negative LTV of −0.039%.Single-stage training performs slightly better on AUD@1d but degrades monetization-related metrics.
  • Backbone scale: Qwen3-8B achieves the best overall backbone performance, with highest LTV improvement of +0.064% and positive gains across the reported long-term metrics.Smaller backbones can underexplore candidate edits, while increasing capacity to 14B is not monotonic.
  • Rollout number: Eight rollouts achieve the highest LTV improvement of +0.050% and consistent gains across all reported metrics.Four and six rollouts underperform, while further increasing rollout count fails to bring additional benefits.
  • LRP behavior: LRP predictions preserve the ordering of ground-truth observations across populations, with mean absolute error of 0.0019% for Low-ST High-LT.The comparison is made across short-term value quantiles and population groups.

6 Conclusion

DMRL combines robust optimizer training with long-term reward modeling for skill optimization under noisy, delayed, and heterogeneous feedback. Its deployment in a large-scale advertising recommendation system demonstrates significant improvements over baselines and supports practical industrial use.

  • Framework: DMRL integrates DRPO for robust optimizer training with LRP for long-term value modeling under population heterogeneity.A two-stage training strategy further stabilizes optimization and reward modeling.
  • Evaluation: Figure 4 compares LRP-predicted and ground-truth LTV across short-term value quantiles and populations.The figure concerns long-term value modeling under heterogeneous user populations.
  • Deployment: DMRL demonstrates significant improvements over baselines after deployment in a large-scale advertising recommendation system.The conclusion presents this deployment as evidence of practical utility in a real-world industrial environment.

A.1 Ethical Considerations

The ethical discussion identifies fairness, privacy, and societal-impact risks in automated advertising optimization and describes monitoring, governance, and rollout safeguards. It also proposes stronger fairness, privacy, and auditing measures as future work.

  • Fairness and bias: Population-aware optimization may favor segments with higher monetization potential or more predictable response patterns.The authors use segment-wise evaluation and constraint-aware monitoring, restricting optimization when significant imbalances appear.
  • User privacy: Offline LRP training on platform data requires privacy-preserving handling through anonymized logs, controlled feature access, and retention policies.Training data access and model artifacts are subject to internal review and management controls.
  • Societal impact: Large-scale automated optimization may overfocus on monetization, potentially affecting user autonomy, advertiser balance, or content diversity.Safeguards include bounded rollout, online anomaly monitoring, and business-rule constraints.
  • Mitigation and future work: Future safeguards include fairness-aware objectives, privacy-enhancing techniques, and more transparent auditing tools for long-term optimization effects.These extend existing segment-wise monitoring, privacy-compliant data handling, and controlled rollout procedures.
Loading 2609.02170v1…