PaperlayerView exact revision on arXiv

Source-linked AI summary

GRP v0.1 Technical Report

Wenfeng Zhuo, Vincent Xue, Charles Wei, Cong Ni, Ruiming Lu, Jiwen Ren, Mo Li, Peng Yang, Xufei Wang, Dongheng Li, Jiacong He, Yi Song, Yufei Fan, Mikhail Obukhov, Yiwen Chen, Yvette Liu, Yin Ye, Chengjie Wu, Mingtao Zhang, Jinchao Ye, Lili Zhang, Chunhui Zhu

arXiv:2609.36688v1cs.IR

TL;DR

Production recommendation cascades remain difficult to replace with a single generative model, particularly when ranking and reward alignment still matter. GRP combines Semantic-ID generation, ranking, and reward modeling, and is deployed progressively. Online tests report gains from replacing weaker sources, while the report identifies gaps in ranking quality and engagement metrics.

  • Problem

    Production cascades are difficult to replace directly because they contain granular optimizations, while generative retrieval still needs ranking and reward models that handle generated candidates.

    Source: §1 · p. 4

  • Method

    GRP combines Semantic-ID generation with a jointly trained ranking module whose frozen weights also provide rewards for reinforcement-learning post-training, and enters production through progressive deployment.

    Source: §1 · p. 4, §3.1 · p. 6

  • Results

    Online tests found GRP near the top on completion and sharing and above the all-source average on almost every engagement axis; replacing weaker sources yielded significant gains in views, view time, and shares.

    Source: §1 · p. 5

  • Takeaways & Limitations

    The report supports evaluating and expanding generative recommendation through measurable deployment steps rather than attempting immediate cascade replacement.

    Source: §1 · p. 4

    The in-model MHP still trails the production ranker, especially on rare engagement heads, and every candidate still passes the late-stage ranker.

    Source: §7 · p. 22

Abstract

from arXiv · show

Industrial recommendation systems rely on multi-stage cascades whose retrieval, ranking, and serving components are difficult to replace jointly. We present GRP, a generative recommendation framework that combines retrieval, ranking, and reward modeling in a single encoder-decoder model, and evaluate a progressive path toward end-to-end recommendation. The model generates multimodal Semantic IDs and scores candidates with a jointly trained ranking module. The frozen ranking module then supplies rewards for reinforcement-learning post-training. We introduce mGRPO, which adds a reference-anchored margin to reward optimization to preserve the likelihood of logged targets. Offline experiments examine history encoding, model capacity allocation, event selection, tokenization, and reward discrimination. Serving optimizations reduce end-to-end retrieval latency by 69%. Online experiments evaluate the model as a retrieval source, with early-ranking bypass, and with replacement of weaker sources. In a retrieval-only comparison, view time increases by 0.46% and shares by 0.77% relative to production. A separate comparison combining bypass and source replacement yields increases of 0.82% in view time and 2.56% in shares, with neutral platform-level guardrails. These results support progressive deployment while identifying remaining gaps in ranking quality and performance across recommendation metrics.

1 Introduction

GRP targets the gap between generative retrieval and production recommendation cascades, where ranking, freshness, and reward alignment remain difficult to replace. It unifies retrieval, ranking, and reward modeling while introducing a progressive deployment path and reporting online gains alongside remaining gaps.

Motivation

  • Production cascades remain difficult to replace because direct head-to-head replacements produced negative results and a bare generator lacks many granular cascade optimizations.These include freshness rules, eligibility filters, calibration layers, and diversity controls.Source: §1 · p. 4
  • Generative retrieval still needs ranking because Semantic IDs can map to multiple items and raw generative ordering is far from a good final ordering.A downstream ranker materially improves view time in the reported system.Source: §1 · p. 4
  • The report identifies a rapidly churning catalog and the difficulty of ranking generated candidates as obstacles to exact-item retrieval and reward alignment.On the short-video surface, the catalog turns over almost completely day-over-day.Source: §1 · p. 4

Our approach

  • GRP first joins production as an additional retrieval source, then expands where incumbent-source comparisons support replacement and eventually ranker bypass.This stages deployment as measurable, individually deployable changes.Source: §1 · p. 4

Contributions

  • GRP couples Semantic-ID generation with a jointly trained ranking module and reuses its frozen weights as the reward model for reinforcement-learning post-training.The ranking module receives detached inputs from the generative trunk.Source: §1 · p. 5
  • Serving optimizations cut end-to-end retrieval-stage latency by roughly two-thirds using KV caching, CUDA-graph capture, C++ preprocessing, and batched reverse lookup.Source: §1 · p. 5

Online results and a per-source gap analysis

  • Online tests report GRP near the top on completion and sharing, above average on almost every engagement axis, and significant gains in views, view time, and shares after replacing weaker sources.Platform guardrails remained neutral.Source: §1 · p. 5

2 A Progressive Path to End-to-End Generative Recommendation

The report stages migration toward end-to-end recommendation as independently measurable, shippable changes evaluated on incumbent metrics. It progresses from adding a retrieval source to replacing weaker sources and selectively bypassing rankers.

2 A Progressive Path to End-to-End Generative Recommendation

  • The migration is staged so each step is independently measurable and shippable, with comparisons made on the incumbent system’s own terms.Source: §2 · p. 5

2.1 Three phases

  • Initially, GRP receives a fixed retrieval quota while its candidates pass through unchanged downstream rankers, enabling same-user, same-metric comparisons with incumbent sources.Shortfalls identify bottlenecks to address before replacement.Source: §2.1 · pp. 5–6
  • Sources GRP dominates on relevant axes can be retired and their quota reassigned; quota grows while auxiliary sources remain for properties such as exploration or freshness.The report reaches this replacement-and-growth phase.Source: §2.1 · p. 6
  • A growing fraction of GRP candidates can bypass early and then late rankers; migration is complete when those rankers no longer change the served slate.The online experiments exercise early-ranking bypass.Source: §2.1 · p. 6

2.2 Measurement protocol

  • The measurement protocol combines short-video surface metrics, platform-level guardrails, and per-source performance measures.Per-source measures include source rate, ranking pass rate, and engagement profile.Source: §2.2 · p. 6

3 Model Architecture and Pre-training

GRP generates Semantic IDs from user context and event history while a jointly trained, detached ranking module scores candidates. Its architecture uses hierarchical item codes, item-level history fusion, and independent block-wise target decoding.

3.1 Overview

  • GRP shares an encoder–decoder trunk between Semantic-ID retrieval and an MHP ranking module that scores candidates and supplies the frozen reward for post-training.Ranking inputs are detached so ranking loss does not perturb the generation trunk.Source: §3.1 · pp. 6–7

3.2 Tokenization: Qwen3-VL Semantic IDs

  • Items use hierarchical Semantic IDs formed by residual quantization of Qwen3-VL vision–language embeddings into three code levels.An auxiliary co-engagement contrastive objective improves codebook utilization and item uniqueness over the prior tokenizer.Source: §3.2 · p. 8

3.3 Generation module (encoder–decoder)

  • The decoder is deeper than the encoder in the reported 1:2 layer configuration, while RoPE is applied only to encoder self-attention.Decoder self-attention is causal within item blocks; RoPE is disabled for decoder self-attention and cross-attention.Source: §3.3 · pp. 8–9
  • The encoder combines user-context tokens with explicit-engagement and watch-event groups, fusing each group’s features with its Semantic-ID embeddings.Source: §3.3 · p. 8
  • Fusing each history item into one encoder token accommodates B_ref items, versus B_ref/4 with SID-level encoding under the same encoder-token budget.The report describes the quality cost as effectively free.Source: §3.3 · p. 8
  • Per-user event sampling reserves a protected budget for scarce explicit actions and allocates slots by event type using a sublinear rule with a floor.This prevents abundant passive watches from crowding out rare valuable actions.Source: §3.3 · pp. 8–9
  • Target items are decoded independently in separate blocks, with causal attention within each item and blocked attention across items.This avoids broadcasting encoder states across all targets or beams until needed.Source: Figure 2 · p. 9, §3.3 · p. 9
  • Generation uses teacher-forced token-level cross-entropy over content-code positions, masking separators and padding.Source: §3.3 · p. 10

3.4 Ranking module (MHP)

The MHP module scores arbitrary candidates with calibrated engagement predictions while keeping ranking gradients out of the generative representation. Its training objective combines generation and ranking losses without gradient-level coupling.

3.4 Ranking module (MHP)

  • MHP scores arbitrary candidates without consuming decoder outputs, making it usable for candidates not generated by the decoder.Source: §3.4 · p. 10
  • Detached candidate, encoder, and user-context inputs ensure ranking loss updates only MHP parameters and does not distort the generative representation.Source: §3.4 · p. 10
  • Each candidate query cross-attends to history and also enters the DCNv2 cross network directly; the residual path prevents the module from ignoring candidates.Source: §3.4 · p. 10
  • MHP trains separate engagement-classification and log-watch-time regression heads, whose calibration matters because the frozen module later supplies the RL reward.Source: §3.4 · p. 10

3.5 Training objective and optimizers

  • The total pre-training objective sums generation and ranking-head losses, but detached MHP inputs prevent ranking gradients from reaching the generation trunk.Dense weight matrices use Muon, 1-D parameters use Lamb, and sharded sparse embeddings use a row-wise adaptive optimizer.Source: §3.5 · p. 11

3.6 Empirical results

Offline evaluation examines history encoding, model capacity, event selection, Semantic-ID tokenization, and freshness on held-out production short-video sessions. The results show that retrieval quality depends on both efficient history use and matching the model’s representable catalog to fresh, high-intent items.

3.6 Empirical results

  • Offline metrics use held-out production short-video sessions from a future date, and each study is a one-factor comparison against its own baseline.The reported metrics include HR@K, Recall@K, and reward-weighted Reward Recall@K.Source: §3.6 · p. 11

3.6.1 Scaling the user-history length with item-level fusion

  • Item-level encoding fits 4× the history at essentially unchanged throughput and raises HR@10 by 1.7% and HR@20 by 0.9%; extending to 2B_ref items halves throughput without improving retrieval.The comparison holds encoder-token length matched; the 4× history fit comes from representing each history item with one fused token.Source: §3.6.1 · p. 11, Table 1 · p. 12

3.6.2 Rebalancing dense capacity toward the decoder

  • Moving the encoder/decoder split from 2:1 toward 1:2 improves HR@5/10/20 and validation loss monotonically while raising throughput by 21%.The reported model uses a 1:2 encoder-to-decoder ratio at fixed total depth and B_ref-item history.Source: §3.6.2 · p. 11, Table 2 · p. 12

3.6.3 Preserving sparse explicit actions in longer histories

  • Merging explicit-engagement events into a protected stream improves reward-recall@20 by 20% and raw recall@20 by 7% versus the item-level baseline.The reported configuration reserves 25% of B_ref for explicit events; scaling the budget past B_ref is flat to down, with 2B_ref slightly worse, attributed to insufficient model capacity.Source: §3.6.3 · p. 12, Table 3 · p. 13

3.6.4 A higher-resolution Semantic ID improves retrieval

  • Replacing MM-SID with Qwen3-VL SID improves recall@20 by 11.9%, reward-recall@20 by 10.3%, and HR@20 by 9.9%.At matched codebook budget, the higher-resolution, non-uniform Qwen3-VL codebook also roughly doubles item uniqueness; exact-tuple matching makes the comparison harder for it.Source: §3.6.4 · p. 13, Table 4 · p. 13

3.6.5 Freshness dominates out-of-window

  • Exact-item retrieval declines sharply within days of the training cutoff as fresh items fall out of vocabulary, while coarse-cluster recall stays essentially flat.Continuous incremental retraining incorporates new items; RL can redistribute probability among represented items but cannot create representations for post-cutoff items.Source: §3.6.5 · p. 13

4 Post-training: Reward-Guided Alignment

GRP uses its frozen ranking module as the reward for on-policy post-training, while mGRPO adds a reference-anchored margin to protect logged-target likelihood. The section addresses reward discrimination between candidates and the risk that reward optimization erodes recall.

4 Post-training: Reward-Guided Alignment

  • Post-training adjusts candidate probabilities using a reward, while continuous incremental retraining addresses the frozen model’s rapid loss of exact-item recall as fresh items leave its vocabulary.Coarse-cluster recall remains essentially flat as exact-item recall declines.Source: Figure 3 · p. 14

4.1 GRPO against a frozen in-model reward

  • The frozen MHP scores temperature-sampled Semantic-ID candidates, and its view-time head is the sole reward in the reported reinforcement-learning comparison.The encoder, embeddings, normalization state, and MHP remain frozen, keeping the reward stationary while the decoder policy is trained against a frozen reference.Source: §4.1 · pp. 14–15

4.2 Making the reward discriminate between candidates

  • The initial reward varied by only 0.005–0.16% within a user’s candidate group because the MHP’s cross-attention largely discarded candidate information.The redesign passes candidate queries directly through the cross network, adjusts capacity, initializes heads from base-rate priors, and clamps classification logits.Source: §4.2 · p. 15
  • Three dense heads gained 0.11–0.13 AUROC and all three reported sparse heads gained 97–184% AUPRC after the MHP redesign.The skip head regressed by −0.018 AUROC, while Recall@20 (+0.0004) and reward-recall@20 (+0.0001) were unchanged; within-user reward spread rose from under 0.2% to a median above 200%.Source: §4.2 · pp. 15–16, Table 5 · p. 16

4.3 mGRPO: stopping reward optimization from collapsing recall

  • GRPO can erode logged-target recall because sampled groups often omit targets, leaving reward-driven exploration without a reliable anchor in observed behavior.A KL penalty slows drift but does not target movement of probability mass away from logged targets.Source: §4.3 · p. 16
  • mGRPO’s reference-anchored margin activates only when a logged target loses competitive position, so it does not oppose reward gains that leave recall intact.The margin is scaled to the target’s position under the reference model.Source: §4.3 · p. 16
  • +1.6% reward-recall@10 with recall@10 at +0.3% on one node, and +3.1% / +0.9% on two nodes, when GRPO and the margin are combined.GRPO alone decreases recall@10 by 1.25% without increasing reward-recall@10; the margin alone preserves recall but yields limited reward movement (+0.5%).Source: §4.3 · p. 17, Table 6 · p. 17

5 Deployment and Serving

The serving path generates and scores Semantic-ID candidates, maps them asynchronously to current items, and allows configurable integration with existing ranking stages. Optimizations target both feed orchestration and model-server latency.

5.1 Serving architecture

  • The online request path runs the user-context encoder, block-wise Semantic-ID decoder, and MHP scorer; an asynchronously refreshed catalog maps generated IDs to eligible items.A single Semantic ID can map to multiple items, which are combined into the request’s candidate pool after quality and eligibility filtering.Source: §5.1 · p. 17
  • A configurable funnel can bypass early ranking or replace selected stages while keeping a production ranker in the loop as the unified model matures.The control surface isolates generative retrieval from downstream ranking and governs how far candidates travel on GRP’s own scores.Source: §5.1 · p. 18

5.2 Reducing end-to-end latency

  • Orchestration changes reduced its p95 contribution by approximately 75% or more, while server-side KV caching and CUDA graphs cut generation-stage latency by approximately 85%.Other changes included reducing request-body p99 size by approximately 67–75%, event-flattening time by approximately 96%, and preprocessing time by approximately 37%; component measurements use different boundaries and are not additive.Source: §5.2 · p. 18, Table 7 · p. 18

6 Online A/B Test

Online tests evaluate GRP as a retrieval source, with early-ranking bypass, and with replacement of weaker sources. The combined bypass-and-replacement comparison improves surface metrics at neutral platform guardrails, while ranking quality remains a deployment boundary.

6 Online A/B Test

  • The tests treat retrieval-source quality, early-ranking bypass, and source replacement as independent deployment axes with separate success criteria.The bypass goal is greater GRP traffic share with neutral topline metrics; the combined replacement test seeks both larger share and a topline gain.Source: §6 · p. 19

6.1 Setup

  • Every treatment adds GRP as an online source on the short-video surface, and platform-level guardrails remain neutral or positive across configurations.The supervised model began 14th by served share with views 0.00%, view time +0.04%, and shares +0.58%, none significant.Source: §6.1 · p. 19, §6.2 · p. 19

6.2 Axis 1: GRP as a retrieval source

  • With a 20× decode budget and no ranking bypass, GRP reached 4th by served share, with +0.46% view time and +0.77% shares against production, both significant.At the smaller budget, view time rose +0.39% (p= .031) and views +0.31% (p= .091), with only the view-time gain significant; RL first lifted view-time contribution by +0.45% over SFT.Source: §6.2 · p. 19, Table 8 · p. 20
  • In the retrieval-source comparison, GRP ranked 2nd on completion and sends and was above the all-source average on favorites, while watch-time and skip ranks remained behind the strongest sources.The comparison uses the early-bypass configuration and all-source average in the same treatment.Source: §6.2 · p. 19, Table 9 · p. 20

6.3 Axis 2: bypassing early ranking

  • Early-ranking bypass raised GRP’s served-share rank from 14th to 4th with post-training and a larger decode budget, but the supervised bypass configuration traded +2.87% shares for −0.45% view time.Post-training brought view time to +0.04% and shares to +0.98%; the larger decode budget then yielded −0.24% view time (p= .177) and +2.30% shares.Source: §6.3 · p. 20, Table 10 · p. 21

6.4 Axis 3: bypass with retirement of weaker sources

  • +0.47% views, +0.82% view time, and +2.56% shares against production when early-ranking bypass is combined with retirement of weaker sources, all significant at neutral platform guardrails.GRP ranked 10th by served share in this experiment, so the gains came with served share holding rather than growing.Source: §6.4 · p. 21, Table 11 · p. 21

6.5 Where the path stands

  • The late-stage ranker remains in the loop, and the in-model ranker still trails production, especially on rare favorite, share, and reply heads.Average watch time and skip rate also remain behind the strongest individual sources.Source: §6.5 · p. 21

7 Conclusion, Limitations, and Future Directions

GRP unifies retrieval, ranking, and reward modeling and advances toward end-to-end recommendation through progressive production deployment. Remaining gaps in ranking quality and recommendation metrics still constrain replacement.

7 Conclusion, Limitations, and Future Directions

  • GRP unifies retrieval, ranking, and reward modeling, and its post-trained early-bypass configuration yields strong engagement while replacing weaker sources produces surface-level gains at neutral guardrails.The report frames progressive migration as the path toward production deployment.Source: §7 · p. 22
  • The path to a fully end-to-end recommender remains incomplete, and the progressive protocol measures that distance directly.The report’s proposed general lesson is to absorb ranking and reward modeling, retrain continuously for catalog non-stationarity, and migrate one measurable source and stage at a time.Source: §7 · pp. 22–23
  • The post-trained early-bypass configuration remains below average on boost rate and trails the strongest individual sources on watch time and skip rate, so auxiliary sources remain necessary.Quota growth is bounded by these gaps; the proposed next step is multi-objective rewards that include those axes.Source: §7 · p. 22
  • The MHP module still trails the production ranker, especially on rare favorite, share, and reply heads, where absolute AUPRC is low and rewards are least informative.The report proposes more capacity and training data for rare heads, plus calibration monitoring against the production ranker.Source: §7 · p. 22
  • The deployed trunk is relatively small: histories beyond B_ref events are flat to negative, while RL gains grow with batch scale.The report proposes evaluating larger dense models and repeating history-length and post-training ladders at each size.Source: §7 · p. 22

B Extended Related Work

Prior work spans Semantic-ID retrieval, end-to-end generative recommendation, and generative modeling for ranking. GRP builds on these strands by combining a jointly trained ranking module, a frozen in-model RL reward, and progressive displacement of the incumbent cascade.

Generative retrieval with Semantic IDs

  • Semantic-ID retrieval work frames retrieval as sequence generation: TIGER introduced Semantic IDs, while PLUM adapts pretrained language models to this formulation at industrial scale.This line establishes tokenization and decoding machinery, but targets retrieval alone.Source: B Extended Related Work · p. 25

End-to-end generative recommendation

  • OneRec, OneRec-V2, UniPinRec, and GPR explore generative systems that unify retrieval and ranking or align recommendations with preference signals.GRP differs by reusing its own jointly trained ranking module, frozen, as the RL reward and by displacing the incumbent cascade progressively.Source: B Extended Related Work · pp. 25–26

Generative modeling for ranking

  • HSTU models ranking and retrieval with a unified action sequence but retains distinct downstream heads, while Gryphon adds a jointly trained scoring head to a Semantic-ID generator.GRP extends this design by reusing that head, frozen, as the reinforcement-learning reward.Source: B Extended Related Work · p. 26

Policy optimization

  • mGRPO builds on GRPO, a critic-free PPO variant, and anchors its margin to the reference model’s competitive gap on logged targets rather than pairwise preferences.This anchoring is what allows it to protect recall specifically.Source: B Extended Related Work · p. 26
Loading 2609.36688v1…