PaperlayerView exact revision on arXiv

Source-linked AI summary

From Interests to Semantic IDs: Retrieval-Grounded Credit Assignment for Generative Recommendation

Mengdan Zhu, Yufan Zhao, Yao Zhao, Sophie Di, Tao Di, Yulan Yan, Sridhar Iyer, Liang Zhao

arXiv:2609.29983v1cs.IRcs.AI

TL;DR

Item-level SID rewards do not evaluate reasoning traces and can assign identical or no group-relative credit. The paper executes generated interests as catalog queries and routes retrieval advantage to target-hitting spans. Across three Amazon Reviews datasets, this improves SID recommendation over a matched control; generated interests also support auxiliary candidate recall and decoding.

  • Problem

    Item-level SID rewards supervise decoded tokens but leave traces unscored and can provide identical or zero group-relative credit.

    Source: §1 · pp. 1–2

  • Method

    The method runs generated interest hypotheses through a frozen catalog retriever and routes retrieval advantage only to target-hitting interest spans, not final SID positions.

    Source: §5 · p. 9

  • Results

    Across three Amazon Reviews datasets, retrieval-grounded credit improves matched-control Recall@10 by 15.5%–39.6% and NDCG@10 by 18.5%–61.5%.

    Source: §4.2 · p. 6, §5 · p. 9

  • Takeaways & Limitations

    Making intermediate predictions executable lets them provide localized supervision without a separate process reward model.

    Source: §5 · p. 10

    Generated traces are optimized for retrieval utility, not verified as faithful explanations, and should not be presented without further validation.

    Source: §6 · p. 10

Abstract

from arXiv · show

Semantic IDs (SIDs) encode each catalog item as a short token sequence, enabling generative recommenders to predict the next item autoregressively. Reasoning-enhanced variants, an increasingly common extension, first generate a textual trace and then decode a next-item SID by beam search. Such recommenders are commonly trained with group-relative policy optimization under an exact-match SID reward, which is sparse in large catalogs. Two failure modes follow. When all rollouts in a group miss the target, the group yields zero advantage and no learning signal. Rollouts sharing the same SID reward receive identical advantages, however much their traces differ. In both cases the reward reflects only the decoded SID, never the reasoning that produced it. This creates a credit-assignment gap. We address this gap with retrieval-grounded query attribution. Each trace is structured into a history summary, a set of interest hypotheses, and a final SID. A frozen retriever executes every hypothesis as a catalog query, so that each hypothesis becomes independently verifiable rather than judged only through the final SID. A rollout is rewarded when any of its queries retrieves the target within the \mbox{top-$K$}, and per-query hit indicators localize that reward to individual hypotheses. Credit is thus assigned at the span level: only hypotheses that individually hit receive positive retrieval advantage, while the retrieval channel never updates the final SID span. Rollouts that share a SID reward can therefore receive different updates. Across experiments on three Amazon Reviews datasets, this yields consistent improvements in SID recommendation. On Video Games, an oracle analysis further reveals the potential of interest-conditioned SID decoding: selecting the target-relevant query among generated interests improves both recall and ranking.

1 Introduction

Item-level rewards can leave reasoning traces without useful credit, even when the final SID reward varies. The paper makes generated interests independently verifiable through retrieval and routes credit to individual queries.

1 Introduction

  • Item-level SID rewards cannot distinguish traces that share a reward, and constant-reward groups yield zero SID-derived advantage.Prefix matching grades decoded item tokens but still does not evaluate the trace itself.Source: §1 · pp. 1–2
  • The method executes formatted interest hypotheses against a frozen retriever and routes positive retrieval advantage only to individually hitting interest lines, never the final SID span.The retrieval signal can still be zero when retrieval rewards are constant within a group.Source: §1 · p. 2
  • When all SID rewards are zero but retrieval rewards differ, retrieval can still update interest lines.In the example, only the second of four queries hits the target, so only that line receives positive retrieval advantage.Source: §1 · p. 2
  • Generated interests can also form a recall pool and optionally constrain candidate-specific SID beam search.An oracle analysis examines selecting the target-relevant generated query for interest-conditioned decoding.Source: §1 · p. 2
  • Across three Amazon Reviews datasets, the proposed method improves SID recommendation.The introduction also highlights an oracle analysis of interest-conditioned SID decoding.Source: §1 · p. 2

2 Related Work

Related work covers semantic-ID recommenders, reasoning for recommendation, process supervision, and language-based retrieval. This paper connects retrieval inputs with SID decoding by using generated interests both for training feedback and inference-time candidates.

2.1 Semantic-ID Generative Recommendation

  • TIGER [15] represents items with quantized content-embedding codes, while later work changes tokenizer learning, adds collaborative signals, or aligns item tokens with language models.The paper retains the base recommender’s tokenizer and SID space, focusing instead on retrieval supervision during reinforcement learning.Source: §2.1 · p. 2

2.2 Reasoning for Generative Recommendation

  • Reasoning-enhanced recommenders use explicit language traces or latent-space computation, but readable text alone does not establish that a trace is useful.Existing methods usually judge traces through the final recommendation; this work also tests whether stated interests retrieve the target.Source: §2.2 · p. 2

2.3 Process Supervision

  • Prior process-supervision methods localize feedback along SID paths or tokens, whereas this method verifies generated language interests through retrieval and routes credit to hitting query lines.Examples include PROMISE [3], HCGRec [30], and SAPO [33].Source: §2.3 · pp. 2–3

2.4 Language-Based Retrieval

  • Language-based retrieval uses generated text as a recall interface, while SID reasoners generate item tokens; this method connects both within one policy.Generated interests can provide candidate recall or support direct SID decoding.Source: §2.4 · p. 3

3 Methodology

Training proceeds through two stages that prepare SID-language alignment and structured reasoning, followed by retrieval-grounded group-relative policy optimization. The final stage applies distinct feedback channels to the response, trace, and individual interest queries.

3 Methodology

  • Training uses two pre-RL stages followed by retrieval-grounded policy optimization, with each checkpoint initializing the next stage.Stage 1 aligns SID tokens with item language and sequential behavior; Stage 2 teaches a history summary and future-interest statements before the SID.Source: §3 · p. 3
  • Stage 3 separately normalizes exact SID-match, rule-based trace-structure, and frozen-retriever rewards, applying advantages to the response, trace, and interest lines.Positive retrieval advantage goes only to interest lines whose queries hit the target.Source: §3 · p. 3
  • At inference, generated interests form an auxiliary recall pool that can optionally define a trie for candidate-constrained SID decoding.Source: §3 · p. 3

3.1 Problem Definition

The problem setup represents catalog items and user histories with fixed Semantic IDs, then defines a structured trajectory containing reasoning and a predicted next-item SID. The supplied passages also specify fixed tokenizer details and trajectory success criteria.

3.1 Problem Definition

  • Each catalog item v has a fixed L-token SID, and the user input concatenates the SIDs of the items in that user’s interaction history.The SID of the held-out next item is the target.Source: §3.1 · p. 3
  • The setup reuses domain-specific three-level RQ-VAE tokenizers, with L = 3 and 256 entries in each codebook, fixed throughout training.SID tokens are added to the language model vocabulary so the autoregressive policy can generate reasoning and the final SID.Source: §3.1 · p. 3
  • A Stage-3 trajectory contains a history summary, a future-interest block, and a predicted SID; success is assessed for trace validity, SID match, and target retrieval.The interest path succeeds when at least one retrieved item has the held-out target’s SID.Source: §3.1 · p. 3

3.2 Pre-RL Initialization

Pre-RL initialization reuses SIDReasoner’s Stage-1 alignment data and adds GPT-4o supervision for a retrieval-query reasoning format.

3.2 Pre-RL Initialization

  • Stage 1 reuses SIDReasoner’s alignment data [6], and Stage 2 uses newly constructed GPT-4o supervision whose reasoning format is adapted to expose retrieval queries.Source: §3.2 · p. 3
  • The domain-specific SID tokens are trained using the eight streams in Table 1.Source: Stage 1: SID Alignment · p. 3

Stage 1: SID Alignment.

The supplied passages describe training streams and a structured reasoning trace with history summaries, future-interest queries, and a target SID.

Stage 1: SID Alignment.

  • The first five rows and general reasoning use completion-only SFT, while two interleaving streams use full-sequence language modeling.All streams jointly update the model, including newly added SID-token embeddings.Source: Stage 1: SID Alignment · p. 3
  • The completion format includes a history-summary block, a future-interests block, and a target SID.The displayed template includes cited SIDs and allows additional entries in each block.Source: Stage 1: SID Alignment · p. 3
  • Each future-interest line states a plausible next interest grounded in cited history SIDs, and only its query text is sent to the retriever.A summary line pairs cited history SIDs with a factual observation; tags locate each line-level token mask.Source: Stage 1: SID Alignment · p. 4
  • An LLM teacher generates the SFT trace from interaction history, item metadata, and held-out next-item metadata, pairing it with the target SID.Stage 2 uses completion-only SFT for one epoch, and that checkpoint initializes Stage 3.Source: Stage 1: SID Alignment · p. 4

3.3 Retrieval-Grounded Policy Optimization

Retrieval-grounded policy optimization combines SID, trace-validity, and query-retrieval rewards, then routes their relative advantages to the response spans each signal evaluates.

3.3 Retrieval-Grounded Policy Optimization

  • Retrieval-grounded optimization combines separately normalized SID, trace, and retrieval rewards across trajectories with query-level attribution within each trajectory.SID credit applies to the full response, trace credit to the reasoning trace, and retrieval credit to attributed interest lines.Source: §3.3 · p. 4

3.3.1 Catalog-Constrained Rollouts.

  • At each SID position, sampling is restricted to valid next tokens in the catalog trie, so every sampled sequence completes to a catalog SID.Tokens outside the current trie node’s valid outgoing tokens have probability zero.Source: §3.3.1 · p. 4

3.3.2 Reward Signals.

  • The SID reward checks exact SID agreement, the trace reward checks structure and history citations, and retrieval rewards a query when the target ranks within top-K.A rollout receives retrieval reward if at least one query retrieves the target; per-query hit indicators are retained for attribution.Source: §3.3.2 · p. 4

3.3.3 Across-Trajectory Credit.

  • SID, trace, and retrieval rewards are standardized separately within each rollout group; a constant reward contributes zero advantage.The three resulting advantages apply respectively to the full response, reasoning trace, and attributed interest lines.Source: §3.3.3 · p. 5, §3.3.4 · p. 5

3.3.4 Token-Level Credit Assignment.

  • For retrieval-positive trajectories, credit is divided among hitting interest lines; for retrieval-negative trajectories, it is divided across all parsed interest lines.Token masks localize retrieval and trace advantages to their corresponding spans, while the final SID span receives only SID advantage.Source: §3.3.4 · p. 5

3.3.5 SID-Inactive Groups

  • When every sampled SID misses the target, retrieval variation can still distinguish trajectories and assign nonzero credit to interest lines whose queries retrieve it.Source: §3.3.5 · p. 5

3.3.6 Policy Optimization.

  • The policy optimization uses token-level probability ratios and KL divergence against a frozen pre-RL policy, with SID tokens evaluated under the trie-restricted action space.The PPO surrogate is dual-clip PPO.Source: §3.3.6 · p. 5

3.4 SID Decoding with Auxiliary Interest Recall

Inference generates a structured trace before catalog-constrained SID beam search; the generated interests can also supply a retrieval pool for optional candidate-constrained decoding.

3.4 SID Decoding with Auxiliary Interest Recall

  • The primary recommendation output is a ranked list of valid SID candidates from catalog-constrained beam search conditioned on the history and generated trace.Source: §3.4 · p. 5
  • Generated interest queries retrieve top-K catalog results whose deduplicated SIDs form an auxiliary recall pool.The pool exposes catalog coverage captured by the generated interests.Source: §3.4 · p. 5
  • The recall pool can define a candidate-specific SID trie for beam search, producing ranked candidates through constrained generation rather than full-catalog SID reranking.Source: §3.4 · p. 5

4 Experiments

Across three Amazon Reviews domains, retrieval-grounded credit improves full-catalog SID recommendation over a matched control. Query-level routing and interest-conditioned candidate pools also reveal additional retrieval-learning signal and potential for SID decoding.

4.1 Experimental Setup

  • Across three Amazon Reviews domains, chronological 5-core splits hold out final and penultimate interactions for test and validation; SID beams are catalog-constrained to ten items.Source: §4.1 · pp. 5–6

4.2 Overall Recommendation Performance

  • Across three domains, Recall@10 rises 15.5%–39.6% and NDCG@10 18.5%–61.5% versus the matched control; our method leads all 12 Table 4 metrics.Source: Table 4 · p. 6, §4.2 · p. 6

4.3 Retrieval Credit and Routing

  • SID rewards yield zero group-relative advantage in 69.1%–80.8% of groups; retrieval reactivates 37.8%, 37.7%, and 19.6% in final windows for Games, Office, and Industrial.Source: Figure 2 · p. 7, §4.3 · p. 7

4.5 Sensitivity to Reward Encoder and Cutoff

  • On Video Games, the 4B reward encoder matches 8B using half the online encoder capacity; SID metrics peak at Ktrain=50 although interest recall rises with larger cutoffs.Source: Figure 3 · p. 8, §4.5 · p. 9

4.3 Retrieval Credit and Routing

  • Hit-query routing is best on all six metrics; versus block routing, Recall@10/NDCG@10 gains are 11.4%/30.5% Games, 6.2%/6.1% Office, and 4.0%/5.7% Industrial.Source: Table 5 · p. 8, §4.3 · p. 8

4.4 Reward Validity and Interest Utility

  • On Video Games, SID Recall@10 is 27.56% with target-retrieving interests versus 4.78% otherwise; 25.91% of targets absent from the SID beam remain interest-retrieved.Source: Table 6 · p. 8, §4.4 · p. 8

4.6 Qualitative Case Study

  • In the illustrated Video Games case, the action-adventure query retrieves the target at rank 1 while the direct SID beam ranks it second; other interests miss.Source: Figure 4 · p. 9, §4.6 · p. 9

4.4 Reward Validity and Interest Utility

  • On Video Games, target-informed oracle selection yields strongest recall and ranking; all-interest pooling slightly exceeds global SID decoding on both metrics.Source: §4.4 · p. 8, Figure 5 · p. 9

5 Conclusion

Retrieval-grounded query attribution routes retrieval feedback to interest spans that retrieve the target while keeping final SID positions under item correctness. Across three Amazon Reviews datasets, the method improves SID recommendation over a matched control and restores learning signals in many SID-inactive groups.

5 Conclusion

  • Retrieval-grounded query attribution executes generated interests against a frozen retriever and routes retrieval advantage only to target-hitting interest spans, leaving final SID positions under item correctness.This addresses item-level rewards that cannot distinguish traces sharing a reward or provide a signal when that reward is constant within a group.Source: §5 · p. 9
  • Across three Amazon Reviews datasets, the method improves SID recommendation over a matched SID-and-trace control and restores group-relative learning signals in many SID-inactive groups.The results also report that hit-aware routing outperforms coarser credit scopes.Source: §5 · pp. 9–10
  • Generated queries complement direct SID beam search, and executable intermediate predictions provide localized supervision without a separate process reward model.Source: §5 · p. 10

6 Ethical Considerations

The experiments use public Amazon Reviews data without collecting new user data, while identifying privacy, bias, and explanation-faithfulness concerns for deployment.

6 Ethical Considerations

  • The experiments use the public Amazon Reviews benchmark and collect no new user data.Source: §6 · p. 10
  • On identifiable logs, inferred future interests may expose sensitive preferences; generated interests may also reproduce popularity and representation biases.Source: §6 · p. 10
  • Biases in catalog metadata and the retriever may narrow user exposure and reinforce filter bubbles.Source: §6 · p. 10
  • The retrieval-optimized trace is not verified as a faithful explanation, so the authors advise against presenting it to users without further validation.Source: §6 · p. 10
  • The authors recommend restricting access to histories, avoiding storage of generated queries, and auditing retrieval rewards across user and item groups.Source: §6 · p. 10
Loading 2609.29983v1…