PaperlayerView exact revision on arXiv

Source-linked AI summary

Separating Thinking from Spelling for Efficient Generative Recommendation

Danil Gusak, Anna Volodkevich, Ivan Oseledets, Evgeny Frolov

arXiv:2610.32980v2cs.IR

TL;DR

Generative recommenders repeatedly run the full backbone to decode candidate identifiers, a cost that can dominate retrieval. Deco reads history into a compact next-item concept and delegates trie-constrained beam search to a light decoder. Across seven datasets, it matches or exceeds full-backbone accuracy while improving throughput under a 50 ms p95 latency budget.

  • Problem

    Full-backbone SID decoding requires O(BK) backbone decode tokens per user, and beam search can dominate generation time as beam width, SID depth, and backbone size grow.

    Source: §3.1 · p. 3

  • Method

    Deco encodes fused history into a compact next-item concept, then uses a light KV-cached decoder conditioned on that concept and each generated prefix to run trie-constrained beam search.

    Source: §1 · p. 2, §3.3 · p. 3

  • Results

    Across seven datasets, Deco matches or exceeds full-backbone baselines and delivers 3.2–7.3× more throughput under a 50 ms p95 latency budget.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Removing the backbone from candidate expansion makes thousand-candidate pools and LLM-scale backbones practical for generative retrieval.

    Source: §5 · p. 8

Abstract

from arXiv · show

Generative recommenders retrieve items by writing their semantic IDs (SIDs) code by code under trie-constrained beam search, an approach now deployed on large web platforms. Standard autoregressive interfaces pay for the SID length K twice: a history of T items enters the backbone as TK code tokens, and every beam-search step runs through the backbone, so returning B candidates costs O(BK) backbone decode tokens per user. The second cost often dominates, and it grows with the backbone, the SID depth and the candidate budget. We introduce Deco (decoding from concepts), which separates reading the history from writing the identifier. The backbone reads each history item as one fused token and condenses the history once into a compact next-item concept; a light KV-cached decoder conditioned on the concept and the generated prefix runs the entire beam search, while the identifiers, the trie and the autoregressive factorization stay unchanged. The decoder approaches full-backbone accuracy with cross-entropy alone, and distillation improves it further. On seven datasets from e-commerce, movies and music streaming, Deco achieves comparable or better accuracy than both a standard generative recommender and the full-backbone teacher with fused inputs, while delivering 3.2-8.6$\times$ their throughput under a 50 ms p95 latency budget in a production-style serving pipeline. Delegating beam search makes larger candidate pools and models practical: Deco generates 1000 candidates faster than its teacher generates 100, and with a 1.5B-parameter LLM backbone, it matches the 69M-parameter teacher's throughput while achieving higher accuracy.

1 Introduction

Generative retrieval’s full-backbone beam search makes candidate spelling a major cost, even after history inputs are fused. Deco keeps preference modeling in the backbone but delegates SID generation to a light decoder, achieving comparable or better accuracy with higher throughput.

1 Introduction

  • FlatGR reads TK history-code tokens, while both flat and fused models still require O(BK) full-backbone decode tokens per user for beam search.At beam width 100, decoding takes up to 91% of generation time, even with a beam-shared history cache.Source: §1 · p. 2
  • Deco fuses the history, produces a shared next-item concept, and delegates the entire trie-constrained beam search to a light KV-cached decoder without changing the SIDs or trie.The decoder is conditioned on the concept and generated prefix; distillation from a full-backbone model improves accuracy.Source: §1 · p. 2
  • Deco combines fused history representations with delegated decoding, supervising every next-item transition in a single backbone pass.The design leaves preference modeling in the backbone while the light decoder handles candidate generation.Source: §1 · p. 2
  • On seven datasets, Deco matches or exceeds full-backbone baselines and delivers 2.8–5.1× FusedGR throughput at batch size 128 and 3.2–7.3× under a 50 ms p95 latency budget.The reported comparisons cover flat and fused histories.Source: §1 · p. 2

2 Related Work

Prior generative-recommendation work develops item representations, recommendation models, and serving optimizations, while this paper targets the repeated computation of candidate decoding. Deco relates to global-local modeling, distillation, and copying, but retains autoregressive trie-constrained search.

2 Related Work

  • Prior generative-retrieval work improves item tokenization, combines generative and dense retrieval, adapts pretrained LLMs, and scales recommenders to industrial traffic; this paper targets repeated decoding computation.Existing full-backbone approaches still use a large model to generate codes, including decoder-based designs that cross-attend to history at every step.Source: §2 · p. 2
  • Serving work includes search optimizations, speculative decoding, autoregressive heads, parallel code generation, order-agnostic identifiers, and discrete diffusion; Deco instead keeps decoding autoregressive and trie-constrained in a light decoder.The decoder uses a fixed-size cacheable next-item concept, and wider beams or deeper SIDs add decoder computation without further backbone calls.Source: §2 · p. 2
  • Deco applies global-local modeling to SID retrieval, uses distillation for accuracy, and adds a history pointer for repeat consumption.The generator and pointer operate in the same SID space, so their mixture factorizes over the trie.Source: §2 · pp. 2–3

3 Method

The method keeps semantic-ID retrieval and trie-constrained beam search intact while separating history encoding from candidate generation. A backbone produces a compact next-item concept, and a light decoder generates identifiers; training can use interaction data alone or add teacher distillation, with a history pointer for repeats.

3.1 Semantic-ID Generation and Its Cost

  • A semantic ID is a K-code tuple from residual quantization, with later codes refining the coarse cluster selected by the first code.The catalog SIDs form a prefix tree whose valid next codes constrain beam search.Source: §3.1 · p. 3

3.2 Fused History

  • Fused history represents each past item with one token instead of K code tokens, reducing the input from TK to T tokens.Training appends each next item’s gold-code prefix under a block-causal mask so targets attend to the history and earlier prefix codes.Source: §3.2 · p. 3

3.3 Decoding from Concepts

  • The backbone’s final history state is mapped to a compact next-item concept, which conditions a decoder that scores valid SID continuations code by code.The concept can be one or a few vectors; the reported default decoder has two layers of width 256 and attends to at most M+K−1 tokens.Source: §3.3 · p. 3
  • Decoding from concepts leaves the SID trie search to a light decoder that uses the history-derived concept and generated prefix, without attending to the history or keeping a backbone cache.The concept is produced by a single backbone pass; beam search reuses the decoder’s own KV cache.Source: §3.3 · p. 3
  • The adapted concept is a complete per-user serving state of 1 KiB in bf16, compared with 0.4–7.8 MiB for a full-backbone model’s history KV cache.The state can be computed in advance when the history changes, though reported timings recompute it for every request.Source: §3.3 · pp. 3–4

3.4 Training the Light Decoder

  • Deco-CE trains the decoder with cross-entropy from interaction data alone, while distillation adds the teacher’s next-code distribution along the gold prefix.The mixed target retains probability mass on the observed code and allocates smoothing mass to trie-valid continuations the teacher finds plausible.Source: §3.4 · p. 4

3.5 Copying from the History

  • A history pointer mixes generator probabilities with attention over past items’ SIDs, allowing the model to represent repeat targets without making the concept store item identities.The mixture factorizes over the trie, so beam search runs unchanged; the pointer is computed once from backbone states and the decoder still does not attend to history.Source: §3.5 · p. 4

4 Experiments

Across seven datasets, Deco combines fused-history encoding with delegated decoding, achieving comparable or better accuracy than generative baselines while improving serving throughput. Its experiments examine the evaluation setup, scaling behavior, and whether a compact concept suffices for accurate retrieval.

4.1 Experimental Setup

  • The seven public datasets vary by domain, scale, history length, and repeat behavior, and evaluation uses a global temporal split with full-catalog Recall@k and NDCG@k.The latest 10% of interactions define test and the preceding latest 10% define validation; efficiency is measured on one H100 using production-style and common eager-PyTorch setups.Source: §4.1.1 · pp. 4–5, §4.1.2 · p. 5
  • The six core models share backbone family, SIDs, and training budget, while varying history representation, delegated decoding, and teacher distillation.The default backbone has 8 layers and width 640; the light decoder has 2 layers and width 256.Source: §4.1.3 · p. 5, §4.1.4 · p. 5

4.2 Results

  • At batch 128, Deco achieves comparable or better accuracy than FlatGR and FusedGR while delivering 5.5–6.5× and 2.8–5.1× their throughput, respectively.Table 2 separates the effects of history fusion and delegated decoding.Source: §4.2.1 · p. 6
  • Deco’s leave-one-out Beauty results are 0.0481 NDCG@10 and 0.0839 Recall@10, retaining its advantage under that evaluation protocol.The primary evaluation instead uses a global temporal split to avoid temporal leakage.Source: §4.1.2 · p. 5, §4.2.1 · p. 6, Table 4 · p. 6
  • On BPC-23, Deco at beam 1000 is faster than FusedGR at beam 100 and recalls 0.267 versus 0.088; its speedup over the teacher rises from 2.7× to 6.8× as SID depth grows from 2 to 8.Most targets leave the beam in the first two SID levels, and later levels cannot restore a pruned path; on Sports, Recall@100 rises to about depth 5 before leveling off.Source: §4.2.3 · p. 7
  • On BPC-23, Deco’s speedup over FusedGR grows from 4.8× at the 69M-parameter model to 16× at 3B, while at 1.5B it matches the 69M teacher’s throughput and is 11% more accurate in NDCG@10.From S to 3B, FusedGR slows down 258×, compared with 12× for Deco; Deco’s decoding rate stays at 58–59k users/s across backbones.Source: §4.2.3 · p. 7, Figure 5 · p. 7

4.2.4 RQ4: Is a Compact Concept Enough?

  • The compact concept is sufficient on low-repeat datasets: richer history access changes NDCG@10 by less than 5% and Recall@100 by at most about 3%, while increasing latency by up to 20%.On repeat-heavy Yambda, a history pointer raises Deco’s NDCG@10 from 0.079 to 0.171; metrics on new targets change by at most 0.012.Source: §4.2.4 · pp. 7–8, Table 5 · p. 8
  • A 1–4-layer decoder stays within about 3% of the default in NDCG@10 and Recall@100, while one layer cuts latency by 20–28%.Distillation raises N@10 over Deco-CE on every dataset; teacher initialization reaches comparable quality in fewer steps but is optional.Source: §4.2.5 · p. 8

5 Conclusion

Deco separates user preference modeling from SID generation: the backbone encodes history into a next-item concept, and a light decoder handles beam search. Across seven datasets, it matches or exceeds full-backbone accuracy while serving 3.2–7.3× more users under a 50 ms p95 latency budget.

5 Conclusion

  • Across seven datasets, Deco matches or exceeds its full-backbone teacher’s accuracy and serves 3.2–7.3× more users under a 50 ms p95 latency budget.The conclusion also states that refreshing the cached concept before requests removes the backbone from online generation.Source: §5 · p. 8

A Additional Results

Additional results show that concept decoding improves the accuracy–throughput trade-off, scales more favorably with larger backbones, and keeps per-user serving state compact. Candidate-budget and decoder-size studies, alongside a distillation comparison, further characterize the design.

Candidate budget

  • On BPC-23, a beam wider than k adds about 3–4% to Recall@100 and 2% to Recall@300, while widening from 300 to 1000 barely changes Recall@100.Table 6 reports relative Recall@k gains against a reference beam width B = k.Source: Table 6 · p. 10, A Additional Results · p. 10

Accuracy against throughput per dataset

  • On all seven datasets, concept-decoder students achieve higher throughput and Recall@100 than their full-backbone teachers.FlatGR-match is faster but loses accuracy, while Deco-CE is as fast as Deco but less accurate.Source: A Additional Results · p. 10, Figure 8 · p. 11

Quality–throughput trajectories

  • From S to 3B, Deco slows 12× against FusedGR’s 258× as backbone size grows, while its NDCG@10 rises.FusedGR buys small accuracy gains with large throughput losses; Deco’s quality–throughput trajectory is more favorable.Source: A Additional Results · p. 10, Figure 9 · p. 11

Per-user serving state

  • Deco’s cached concept stays at 1 KiB per user, while a full-backbone history KV cache grows with history length and tokenization.The comparison excludes beam-local decoder caches and optional copy state.Source: A Additional Results · p. 10, Figure 10 · p. 11

Decoder size

  • Across a sixfold range of decoder parameters, NDCG@10 and Recall@100 stay within about 3% of the default, while latency grows with depth and width.The study varies decoder depth and width; the table reports batch-128 latency relative to the default.Source: A Additional Results · p. 10, Table 7 · p. 11

Distillation without the concept decoder

  • On Sports and BPC-23, distillation raises NDCG@10 by 19.4% vs. 7.5% and 2.3% vs. 0.8% for Deco vs. FusedGR + KD, respectively.The comparison isolates distillation into a FusedGR student from the concept-decoder student.Source: A Additional Results · p. 11, Table 8 · p. 11

B Tied and Untied Code Tables

The paper considers tying the SID code-vector tables used for reading history, conditioning the decoder, and scoring output codes. Tying improves Recall@100 for FlatGR, FusedGR, and Deco on both datasets, and Deco remains the most accurate model with either tied or untied tables.

B Tied and Untied Code Tables

  • Code vectors can represent history items, condition the decoder on generated prefixes, and score next-code output classes.The main experiments use separate tables for these roles, allowing different learned representations for reading and writing.Source: B Tied and Untied Code Tables · p. 11
  • Tying the tables raises mean Recall@100 for FlatGR, FusedGR, and Deco on both datasets, while Deco is most accurate among the four models with either tied or untied tables.The effect of tying on Deco-CE is mixed.Source: B Tied and Untied Code Tables · p. 12, Table 9 · p. 12
  • Untied fused-model history-side vectors barely overlap with nearest code clusters, whereas output class vectors recover content neighborhoods above chance.The reported output-vector neighborhood overlap is 8–11% on Sports and 39–41% on Yambda; chance is 0.5–0.6%.Source: B Tied and Untied Code Tables · p. 12, Table 9 · p. 12

C Further Implementation Details

The implementation uses a common light decoder across backbone families and specifies training and timing procedures for the reported efficiency results. The serving measurements include backbone prefill and full beam search, while excluding data loading and SID-to-item mapping.

C Further Implementation Details

  • The light decoder uses two layers, width 256, four heads, and M = 2 across backbones; decoder-size experiments vary depth from one to four layers and width from 128 to 512.The default scratch backbone is L; Qwen2.5 backbones read the same fused item tokens as input embeddings.Source: C Further Implementation Details · p. 12, Table 10 · p. 12
  • Training runs for at most 5k–60k optimizer steps depending on dataset, with early stopping on validation NDCG@10.Scratch-model learning rates are selected per family and dataset in the range 10^-4–10^-3; distillation temperature is 1, or 2 on Beauty and MovieLens.Source: C Further Implementation Details · p. 12
  • Efficiency results use one H100 80GB GPU with bf16 weights, and latency covers backbone prefill plus full trie-masked beam search but excludes data loading and SID-to-item mapping.For the 50 ms p95 budget, Deco serves 3.2–7.3× more users than FusedGR and 6.5–8.6× more than FlatGR across seven datasets.Source: C Further Implementation Details · p. 12

Related papers

Loading 2610.32980v2…