PaperlayerView exact revision on arXiv

Source-linked AI summary

KL-MOPD: Discrepancy-Aware Multi-Objective On-Policy Distillation for Generative Recommendation

Weiqin Yang, Yandong Wang, Yuan Gao, Hao Deng, Jinxin Hu, Jing Cao, Jiawei Chen

arXiv:2610.36973v1cs.IR

TL;DR

Generative recommendation must coordinate objectives, but item-wise methods lack token-level guidance and vanilla MOPD weights teachers equally despite token-varying distillation difficulty. KL-MOPD weights teacher supervision by initial student-teacher token-wise KL divergence and improves individual and joint recommendation performance across benchmarks.

  • Problem

    Vanilla MOPD gives teachers equal weights despite token-specific differences in distillation difficulty that shift across decoding steps, limiting balanced token-wise learning.

    Source: §2.3 · p. 5

  • Method

    KL-MOPD weights each teacher's token-wise supervision in proportion to its initial student-teacher token-wise KL divergence.

    Source: §1 · p. 3, §3.1 · p. 5

  • Results

    Across three public benchmarks, KL-MOPD averages +4.91% over the strongest baseline on individual objectives and +4.12% over the strongest baseline and single-objective teachers on the joint objective.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Under mild conditions, KL-MOPD strictly outperforms vanilla MOPD in the additive joint recommendation objective.

    Source: §1 · p. 3

    The theoretical formulation assumes nonnegative bounded objectives and an additive joint objective, though weighted sums can be converted to that form.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Coordinating multiple recommendation objectives (e.g., click and conversion) is critical for real-world generative recommendation (GR). However, since a positive item in GR can be retrieved only when its tokens are successively selected at every decoding step, traditional item-wise methods lack fine-grained token-wise supervision to guide intermediate steps. Motivated by its success in foundation models, we consider introducing multi-objective on-policy distillation (MOPD) into GR to distill two single-objective teachers into a student at each token. Yet, we find that in vanilla MOPD, (i) teachers exhibit unequal distillation difficulty at each token (i.e., token-level imbalance), and (ii) the harder-to-distill teacher varies across tokens (i.e., cross-token inconsistency), leaving each objective poorly learned at certain steps and hindering positive item retrieval for both objectives, leading to suboptimal performance. To address this challenge, we propose KL-MOPD (KL-Weighted Multi-Objective On-Policy Distillation), a simple yet effective distillation method that allocates token-wise teacher weights proportional to the initial student-teacher token-wise KL divergences, thereby strengthening supervision for the harder-to-distill teacher at every decoding step. Despite its simplicity, we theoretically prove that KL-MOPD strictly outperforms vanilla MOPD in the additive joint recommendation objective under mild conditions. Moreover, KL-MOPD is plug-and-play and computationally efficient, requiring only a one-time pre-training weight calibration with negligible training overhead. Extensive experiments on three public benchmarks, a billion-scale industrial dataset, and online A/B tests on our e-commerce advertising platform demonstrate that KL-MOPD significantly improves both individual and joint recommendation performance. The anonymous code is provided for peer review.

1 Introduction

Generative recommendation must coordinate multiple objectives through token-by-token generation, but traditional item-wise approaches lack token-level supervision. KL-MOPD weights teachers by initial student-teacher token-wise KL divergence to address imbalance in vanilla MOPD, with theoretical and empirical support.

1 Introduction

  • Traditional multi-objective methods operate at the item level, leaving GR without fine-grained supervision at intermediate decoding steps.Because a positive item is retrieved only if its tokens are selected successively at every step, missing a target token prevents retrieval.Source: §1 · pp. 1–2
  • Vanilla MOPD gives teachers equal weight despite unequal token-level distillation difficulty, and the harder-to-distill teacher can change across decoding steps.On TencentGR, the value teacher is harder over the first three tokens, while the click teacher is harder over the last two.Source: §1 · p. 2
  • KL-MOPD weights each teacher in proportion to its initial student-teacher token-wise KL divergence, strengthening supervision for the harder-to-distill teacher at each token.The method reduces decision errors for the poorly learned objective at each token, making correct token selection at every step more likely.Source: §1 · p. 3
  • KL-MOPD strictly outperforms vanilla MOPD on the additive joint objective under mild conditions and reduces average item-wise KL divergence to both teachers by 13.70%.It also achieves more balanced teacher fitting at every token.Source: Figure 1 · p. 2, §1 · p. 3
  • KL-MOPD can be integrated by reweighting the token-wise distillation loss, with weights precomputed once and negligible training overhead.It requires neither auxiliary models nor architectural modifications and adds no extra training forward passes.Source: §1 · p. 3
  • KL-MOPD improves individual and joint recommendation on public benchmarks, and reports industrial and online gains in e-commerce advertising.On three public benchmarks, it averages +4.91% on individual objectives and +4.12% on the joint objective versus the strongest baseline; the industrial joint metric improves 3.19%, while online revenue and GMV rise 1.63% and 2.20%.Source: §1 · p. 3

2 Preliminaries

Section 2 formulates generative recommendation as token-by-token item generation with click and value objectives, then introduces on-policy distillation and identifies weaknesses in equal-weight MOPD.

2.1 Task Formulation

  • A positive item is retrieved only when the recommender selects the appropriate token at every autoregressive decoding step.A decision error at any step can divert the generated prefix and cause retrieval failure.Source: §2.1 · p. 3
  • The task combines click engagement and transactional value under an additive joint objective, assuming both objective functions are non-negative and bounded.The value objective can represent conversion, GMV, or eCPM.Source: §2.1 · p. 3

2.3 The Pitfalls of MOPD in GR

  • KL-MOPD produces more balanced token-wise KL divergences and lower item-wise KL divergence to both teachers than vanilla MOPD.Table 1 reports results for TIGER on TencentGR and YooChoose.Source: Table 1 · p. 4

2.2 Multi-Objective On-Policy Distillation

  • MOPD trains a student from fixed click- and value-specialized teachers using student-sampled trajectories and equal teacher weights at each decoding step.The teachers provide next-token distributions conditioned on each student-generated prefix.Source: §2.2 · p. 4
  • Unlike item-wise alignment, MOPD directly aligns token-wise distributions at every step, while on-policy learning operates in the student’s own generation space.The paper states that this on-policy objective is equivalent to maximizing the KL-regularized additive joint objective.Source: §2.2 · p. 4

2.3 The Pitfalls of MOPD in GR

  • Vanilla MOPD’s equal weights leave the initially harder teacher under-fitted, and that teacher shifts across tokens, risking retrieval errors for both objectives.On TencentGR, the value teacher is harder over the first three tokens and the click teacher over the last two.Source: §2.3 · p. 5

3 Methodology

KL-MOPD adapts multi-objective on-policy distillation to token-level differences in teacher difficulty. It weights teachers by their initial student–teacher KL discrepancies, with theoretical and practical advantages described for joint recommendation.

3.1 KL-Weighted On-Policy Distillation

  • The method measures each teacher's distillation difficulty using expected token-wise KL divergence from the initial student.The paper links a larger discrepancy to greater difficulty distilling that teacher.Source: §3.1 · p. 5
  • KL-MOPD assigns each teacher a normalized token-wise weight proportional to its initial student–teacher KL divergence.This gives greater weight to the harder-to-distill teacher at each token.Source: §3.1 · p. 5
  • Stronger supervision for the harder-to-distill teacher is intended to mitigate token-level imbalance and cross-token inconsistency.The authors say this makes item-level decoding failures from under-distilling a teacher at certain tokens less likely.Source: §3.1 · p. 6

3.2 Analysis of KL-MOPD

  • When S_KL > D_T L^3/β^3, KL-MOPD is guaranteed to strictly outperform vanilla MOPD on the additive joint objective.The paper states that this condition is readily satisfied in practice, yielding a strictly positive joint-objective gain.Source: §3.2 · p. 6
  • KL-MOPD requires only one-time precomputation of token-wise discrepancies and weights, with no additional forward passes or memory overhead during distillation.The method replaces uniform weights and is designed to integrate into existing MOPD pipelines and GR backbones.Source: §3.2 · p. 7

4 Experiments

Across public benchmarks, KL-MOPD improves individual and joint recommendation performance; industrial evaluations report offline and online gains. The experiments also compare weighting variants and training efficiency, using multiple datasets, backbones, and recommendation metrics.

4.1 Experimental Setup

  • The experiments use three public datasets and a billion-scale industrial advertising dataset, evaluating two GR backbones against the initial student, both teachers, and eight baselines.The evaluation reports individual objectives and their additive joint objective; public datasets use chronological 70/20/10 splits, while Industry uses its last two days for fine-tuning and evaluation.Source: §4.1 · p. 7

4.2 Experimental Results

  • KL-MOPD achieves the best joint objective u_cv in all public benchmark settings, averaging +4.12% over the strongest baseline and single-objective teachers.It also outperforms all baselines on individual objectives u_c and u_v by an average of +4.91%.Source: §4.2 · p. 8
  • KL-MOPD consistently outperforms item-wise KL, token-wise MSE, and top-16-token KL weighting variants in the TIGER ablation.The variants use token-shared item-wise KL weights, token-wise mean squared probability discrepancies, or KL restricted to the initial student's renormalized top 16 tokens.Source: §4.2 · p. 8, Table 4 · p. 8
  • KL-MOPD precomputes its token-wise weights once before training, adding virtually no overhead over MOPD and offering a better accuracy-efficiency trade-off than TrustMOPD and RCE-KD.Figure 3 compares the additive joint objective with training time per step on TencentGR and YooChoose using TIGER.Source: Figure 3 · p. 7, §4.2 · p. 8
  • On the billion-scale Industry dataset, KL-MOPD outperforms vanilla MOPD and both teachers on single- and joint-objective metrics, averaging +3.19% on u_cv.Table 5 reports offline results for top-100 and top-200 recommendations and compares KL-MOPD only with MOPD.Source: §4.2 · p. 8, Table 5 · p. 8
  • In the real-world industrial e-commerce advertising platform’s online A/B test, KL-MOPD achieved statistically significant (p<0.05) uplifts of +1.63% in advertising revenue (REV) and +2.20% in GMV.Source: §4.2 · p. 8, Table 6 · p. 8

5 Related Work

Prior work addresses multi-objective recommendation and distillation, but much of it operates at the item level rather than across GR's token sequence. KL-MOPD builds on on-policy token-wise supervision while adapting teacher weights to token-level discrepancies.

Generative Recommendation

  • GR represents items as sequences of discrete semantic tokens, making accurate decoding decisions at every token necessary for retrieving a positive item.The paper positions balanced token-wise optimization as addressing this specific property of GR.Source: §5 · p. 8

Multi-Objective Optimization

  • Traditional multi-objective methods balance losses or gradients, or transfer capabilities through rankings, representations, or scores, but operate at the coarse item level.These approaches were designed for recommenders that score whole items, unlike GR's multi-token generation.Source: §5 · p. 8

Recommendation Distillation

  • Recommendation distillation has transferred teacher capabilities through rankings, hidden representations, and prediction scores, mostly at the item level.On-policy distillation for GR provides token-wise supervision, which KL-MOPD retains while addressing multi-teacher imbalance and cross-token inconsistency.Source: §5 · pp. 8–9

On-Policy Distillation

  • Recent MOPD approaches use reinforcement learning or dynamic weighting, while KL-MOPD instead weights teachers by initial student-teacher discrepancies at each token.The paper characterizes these prior approaches as mostly intuitive and not tailored to recommendation, and contrasts them with KL-MOPD's theoretical guarantees over vanilla MOPD.Source: §5 · p. 9

6 Conclusion and Future Directions

The paper concludes that KL-MOPD addresses token-level imbalance and cross-token inconsistency with discrepancy-based teacher weighting, supported by theoretical guarantees and evaluations from public benchmarks through production tests.

6 Conclusion and Future Directions

  • KL-MOPD weights teachers by initial student-teacher token-wise KL divergences and strictly outperforms vanilla MOPD on the additive joint objective under mild conditions.The paper reports validation on public benchmarks, a billion-scale industrial dataset, and production online A/B tests; future directions include extending theory to three or more teachers and generalizing to language tasks.Source: §6 · p. 9

A Proof of Theorem 3.1

The appendix outlines the proof of Theorem 3.1, introducing notation and preliminary results before presenting the main proof.

A Proof of Theorem 3.1

  • The proof proceeds from notation and preliminary results in Section A.1 to the main argument in Section A.2.This section provides the complete proof of Theorem 3.1.Source: A Proof of Theorem 3.1 · p. 11

A.1 Preliminary Results

This appendix establishes notation for request-and-rollout expectations and conditional objectives, then gives preliminary results for optimal students and exponential reweighting. With token-wise teacher weights, the optimal target is a normalized geometric mixture of the teachers.

Notation Simplification

  • For any policy and function, the appendix writes expectation over user requests and policy rollouts as E_X,π[f], and defines L as max{L_c, L_v} and ∅ as the empty prefix.It also introduces oscillation as the maximum minus minimum of a function on a finite set.Source: §A.1 · p. 11

Optimal Students with MOPD

  • The optimal token-wise target policy under weighted KL distillation is the normalized geometric mixture of the two teachers, and its item-wise policy multiplies token probabilities across decoding steps.KL-MOPD and vanilla MOPD correspond to discrepancy-proportional and uniform token-wise weights, respectively.Source: §A.1 · p. 11

Conditional Expected Objective, Increment, and Variance

  • Under the initial student, the conditional expected objective changes after each observed token; its expected token-wise increment is zero.The conditional objective begins as an expectation over complete student rollouts and equals the objective at the completed sequence.Source: §A.1 · p. 11

A.2 Main Proof of Theorem 3.1

The proof relates initial student-teacher token-wise KL divergences to objective-increment variances, then uses these approximations to bound the joint-objective difference between KL-MOPD and vanilla MOPD. It establishes strict improvement when the stated discrepancy condition holds.

Lemma A.2

  • At each decoding step, the initial student-teacher token-wise KL divergence is approximated by the corresponding objective-increment variance.This relation connects the weighting signal to the variance terms used in the joint-objective proof.Source: §A.2 · p. 11

Lemma A.3

  • The proof approximates the joint-objective difference using a leading term involving the difference between the objectives’ increment variances, with a bounded remainder.Substituting the KL-proportional weights yields the theorem’s lower bound.Source: §A.2 · pp. 11–12

Proof of Lemma A.3

  • To obtain the approximation, the proof expands the joint objective under weighted MOPD, approximates an inner expectation by covariance, and bounds the error from shifting the prefix distribution.The weight-independent cross-covariance term cancels when the KL-MOPD and uniform-weight expansions are subtracted.Source: §A.2 · pp. 12–13

Theorem 3.1

  • When S_KL > D_T L^3/β^3, KL-MOPD is guaranteed to strictly outperform vanilla MOPD on the additive joint recommendation objective.The proof derives a lower bound for the improvement between their optimal policies.Source: §A.2 · p. 13

Related papers

Loading 2610.36973v1…