PinGER: Token-Efficient Generative Retrieval with Semantic IDs

1 Introduction

• SID retrieval faces a context-versus-supervision trade-off, beam costs that grow with beam width, and beam collapse onto shared prefixes.Naive compression reduces SID supervision, while temperature sampling lowers spread and produces more nonexistent SIDs.Source: §1 · p. 2
• PinGER trains on every SID token but serves histories with one dense token per item, computing the same function while avoiding the L-token input cost.Its asymmetric mask prevents later positions from reading past SID tokens; the serving stack also constrains decoding to valid SIDs and resolves them from a host-local index.Source: §1 · p. 2
• Online, replacing PinRec with PinGER raises saves by 0.62% and saves of fresh Pins by 3.60%.At matched load, serving is 18 to 31 ms faster at the median than PinRec.Source: §1 · p. 2
• A hierarchical prefix penalty diversifies beams, while cascade attention reads shared history once per request rather than once per beam.The reported serving stack also uses constrained decoding and a host-local index for candidate resolution.Source: §1 · pp. 2–3
• PinGER’s beam span 7.9× more distinct level-1 prefixes than greedy decoding at the deployed penalty.The paper also reports 10.9× faster decoding than paged attention at W=30 and 32× at W=400, with 12.6× less KV memory.Source: §1 · pp. 2–3

2 Related Work

• Embedding GR reads and predicts one dense embedding per item but retrieves through an ANN index, while SID GR generates hierarchical SIDs and decodes directly into items.Hybrids either retain extra serving positions, train on one label per item, or use two large decoders.Source: §2 · p. 3
• Prior acceleration and shared-cache methods still make each beam read the full history; PinGER folds beams into the query dimension to read it once per request.It also constrains decoding with STATIC’s vectorized trie and resolves items from a host-local index.Source: §2 · p. 3
• Unlike flat diverse beam search, PinGER penalizes candidates according to the depth of their shared SID prefix with already chosen beams.The paper reports that temperature, the usual diversity knob in generative retrieval, lowers both spread and validity.Source: §2 · p. 3

3.1 Background and problem setup

• Retrieval generates an ordered candidate list from a user’s chronological interaction history, representing each item with a dense embedding and a multi-code SID.The generated identifiers are decoded with beam search and resolved to items for a downstream ranker.Source: §3.1 · p. 3

3.2 Semantic IDs

• SIDs are quantized from content embeddings, while distillation incorporates engagement information without making SIDs change when the graph embedding is updated.The tokenizer uses five codebooks of size 2048 and distills OmniSage into the quantized representation while retaining the content embedding as its input.Source: §3.2 · p. 3

3.3 Interleaved sequence modeling

• PinGER keeps dense input tokens and SID output tokens in one causal backbone rather than separating them into an encoder and decoder.This design retains a long history in the input while allowing direct SID generation without an ANN index.Source: §3.3 · p. 3

3.3.1 Interleaved layout.

• During training, each item contributes its SID tokens followed by a dense token, with positions indexed by item rather than token.Figure 2 contrasts the interleaved training sequence with serving, where past SID tokens are dropped.Source: §3.3.1 · pp. 3–4, Figure 2 · p. 4

3.3.2 Asymmetric mask.

• The asymmetric mask confines dense queries to dense history and SID queries to dense history plus their own SID prefix, so past SIDs can be dropped at serving without changing the next-identifier distribution.No position outside an item attends to its SID tokens; serving retains one dense token per history item and the SID tokens currently being decoded.Source: §3.3.2 · p. 4

3.3.3 Supervision.

• Training applies weight-tied SID cross-entropy at every code level, conditioning each prediction on dense history and the true shallower codes of its item.Dense positions are not prediction targets, yielding L predictions per history item in a forward pass.Source: §3.3.3 · p. 4

3.3.4 Sparsity-aware attention kernels.

• Custom attention kernels exploit the mask’s dense-history columns and within-item SID triangles; together with dropping the final non-target SID input, they raise training throughput by 94% over plain TIGER at the same context length.The kernels use unmasked attention for shared earlier dense tokens and masked attention within each tile; their attention cost is about L^2/(L+1) lower than TIGER’s, and serving uses standard causal attention.Source: §3.3.4 · p. 4

3.4 Feature embeddings and goal conditioning

• Item features combine content and graph embeddings with SID fusion, while goal conditioning at every SID level steers generation toward the requested action, surface, and time.SID fusion adds a discrete view of the item to its dense representation without extra input positions; diversity penalty spreads the beam across items of the requested kind.Source: §3.4 · pp. 4–5

3.5 Decode-time steering: prefix diversity

• At W=100, ordinary beam search spans only 11.9 distinct level-1 codes because accumulated scores favor continuations of already selected prefixes.The prefixes are semantic, so beams retained within a few subtrees are also similar in content.Source: §3.5 · p. 5
• Temperature sampling does not restore spread: spread and lookup@L both degrade as T rises, and final-level sampling can produce invalid identifiers.Improbable sampled codes are outscored at the next level, returning beam slots to dominant subtrees; the final level has no later pruning step.Source: §3.5 · p. 5
• The hierarchical penalty selects beams one at a time and penalizes parents according to already selected beams and shared-prefix depth.Deeper shared prefixes receive larger penalties; λ=0 recovers top-W selection, and updates use W additions per selected beam.Source: §3.5 · pp. 5–6

3.6 Post-training: training-time steering

• Next-token training on engaged items lacks a contrastive negative signal and an item-level objective aligned with autoregressive inference, leaving retrieval poorly aligned with ranker scores.The paper therefore post-trains with DPO.Source: §3.6 · p. 6
• Preference pairs come from serving logs by contrasting each slate’s top three and bottom three ranker-utility items; DPO is anchored to the pretrained model to avoid recall collapse.The anchor makes the loss stop pushing pairs already ordered correctly by the pretrained model; the DPO term alone causes rapid drift and recall collapse.Source: §3.6 · p. 6

4 Serving Infrastructure

• The serving stack retains Triton inference, feature fetch, batching, and ensemble routing, adding cascade decode, constrained decoding, and host-local SID resolution.The existing PinRec and UniPinRec deployment stack supplies the unchanged serving components.Source: §4 · p. 6

4.1 Cascade decode

• Cascade decode shares each request’s history pass across beams and attends separately to each beam’s short generated-token suffix.Combining the two attention outputs and log-sum-exps reproduces attention over the full sequence.Source: §4.1 · pp. 6–7
• At W=400, cascade attention is 32× faster than paged attention and uses 48.7× less KV memory; its memory stays lower as beam width widens.At W=30, cascade holds 12.6× less KV memory than paged attention; at W=10, it is 4.25× faster.Source: Figure 5 · p. 6, §4.1 · p. 7

4.2 Constrained decoding

• GPU-resident trie constraints mask codes absent from the corpus, so every beam resolves at level 5 without host round trips during decoding.The constraint adds 0.30 ms at W=100 and 0.59 ms at W=400; the trie occupies 168 MiB for 10 M items and 1,368 MiB for 100 M items.Source: §4.2 · p. 7

4.3 In-memory serving

• A host-resident, daily rebuilt index resolves SIDs through vectorized binary search; short fine-level results fall back to coarser prefixes, with candidates interleaved round-robin.Memory-mapped read-only indices share one physical copy per host across replicas.Source: §4.3 · p. 7

4.4 End to end: the composed system

• PinGER is faster than PinRec at every tested beam width; the remote nearest-neighbor call costs 31 ms at p99, while host-local resolution stays under 20 ms at p99.The comparison uses the same serving stack, user sequence length, and matched per-host load.Source: §4.4 · p. 7

5.1 Shared setup

• The shared setup pairs historical cross-surface actions with target-surface sessions, evaluates on later held-out users, and reports Recall@k, prefix diversity, and identifier lookup.The offline recall metrics use beam width 100; the backbone has four transformer layers and SID length L=5.Source: §5.1 · p. 7

5.2.1 Main comparison.

• At the same 1,024 serving tokens, PinGER’s Recall@100 is 9.5% higher than TIGER’s and 3.9× PinRec’s; matching TIGER’s recall takes 170 tokens.PinGER reads six times fewer serving tokens than TIGER; both models gain about 10% Recall@100 as training context grows from 1,024 to 6,144 tokens, without saturation.Source: Table 2 · p. 8, §5.2.1 · p. 8

5.2.2 Ablations.

• Offline ablations show SID fusion reduces Recall@100 by 27.0% when removed, while goal conditioning’s removal costs 17.0%.Removing PinCLIP leaves recall essentially unchanged (Δ≈0), while removing OmniSage costs 1.2%.Source: §5.2.2 · p. 8

5.2.3 Diversity penalty.

• The prefix-diversity penalty raises distinct level-1 prefixes from 11.9 under greedy decoding to 93.6 at λ=3, while top-10 lookup barely moves and recall falls modestly.At T=1.5, temperature sampling yields 3.5 distinct prefixes and 13% top-10 lookup versus 74% under greedy decoding; Recall@100 falls 43% at T=0.7 and 85% at T=1.5, versus 28.5% for the mildest penalty.Source: Table 3 · p. 8, §5.2.3 · p. 8

5.2.4 Post-training.

• DPO post-training lifts offline recall by 2.3% and improves the engaged-versus-impression-only log-likelihood difference by 86%.Online, retrieved identifiers show a 10% improvement in candidate survival rate in the final slate.Source: §5.2.4 · p. 8

5.3 Online A/B testing

• Both online A/B tests compare PinGER with PinRec at a fixed candidate budget, and all reported lifts are statistically significant at the 5% level.The tests run on Related Pins and notifications.Source: §5.3 · p. 8

5.3.1 Related Pins.

• Related Pins saves rise by 0.62%, site-wide saves by 0.61%, grid clicks by 0.94%, and saves of fresh Pins by 3.60%.The experiment also reports a 0.7% latency reduction; the authors attribute fresh-Pin gains to content-derived SIDs being available before engagement embeddings stabilize. Segment analysis reports 5.4% higher save rate for top-decile Pins, 13% for undervalued Pins, and 12% more saves among the most active users.Source: §5.3.1 · pp. 8–9, Table 4 · p. 9

5.3.2 Notifications.

• Notifications show consistent engagement gains with λ=1.0; a stronger λ=3.0 yields a smaller 0.76% click-rate gain and raises unsubscribes.The deployed milder penalty reflects the reported trade-off between diversity and a short notification slate.Source: §5.3.2 · p. 9

• The interleaved layout trains with SID supervision at every position while serving histories with one token per item.The serving stack is built for wide-beam generation and removes nearest-neighbor search from the request path.Source: §6 · p. 9
• Future work includes sharing an SID-based backbone between retrieval and ranking and applying the interleaved layout to LLM-based recommenders.The cited motivation is that multi-token item representations limit the history that fits in LLM context.Source: §6 · p. 9

• Table 5 compares Recall@100 across context lengths and reports each model’s required serving length.The supplied caption gives the sweep’s scope but not its values.Source: Table 5 · p. 10
• Table 6 reports constrained-decoding latency and resident-trie memory on one A10G, with uniform-random corpora as upper bounds.Source: Table 6 · p. 10
• Table 7 compares PinGER and PinRec by segment, reporting changes in impression share and save rate.The caption also specifies the lowest-decile-of-saves and top-by-saves segment definitions.Source: Table 7 · p. 10, A Additional detail · p. 10

• An MLP maps x to latent z_e, and residual quantization produces discrete codes and a quantized vector.The RQ-VAE objective is used, with codebooks updated by gradient descent and a commitment term weighted by β.Source: §A.1 · pp. 10–11
• Engagement distillation keeps x as the tokenizer input while matching neighborhoods of OmniSage embeddings through a head over z_q.The method matches temperature-scaled cosine-similarity neighborhood distributions with KL divergence; its loss must flow into the codebooks.Source: §A.1 · p. 11
• Progressive masking samples a cutoff k and forms z_q from the first k residual levels, while low-count codes are reinitialized from batch encoder latents.The cutoff is sampled uniformly from 1 to L, and the code-count threshold is an EMA below one.Source: §A.1 · p. 11

A.2 Attention cost of the training kernels

• The kernels compute about (L+1)|H|^2/2 attention scores for |H| items.Source: §A.2 · p. 11
• TIGER computes about L^2|H|^2/2 attention scores because each token attends to all codes of earlier items.Source: §A.2 · p. 11
• Under the stated mask, FlexAttention computes about (L+1)^2|H|^2/2 scores, while the kernels’ cost is lower than TIGER’s by a factor of L^2/(L+1), despite an extra token per item.Source: §A.2 · p. 11

A.3 Timestamp gaps in the interleaved layout

• At an item’s first SID position, ψ encodes the true timestamp gap t_i−t_(i−1).Source: §A.3 · p. 11
• At the other SID positions, the predecessor is in the same item, so the gap is zero.Source: §A.3 · p. 11
• The dense token copies the first SID position’s encoding, matching serving where it follows the previous item’s dense token.Source: §A.3 · p. 11

A.4 Computing the prefix penalty

• Each parent’s codes are sorted once per level, while only the counts n_w change across the W selections.Source: §A.4 · p. 11
• Selecting winner w★ increments only n_w★, so the linear penalty update adds column m(·,w★) using W additions without storing the W×W matrix.Source: §A.4 · p. 11
• Because the W steps depend on one another and must run sequentially, each level’s loop runs as one fused GPU kernel.Source: §A.4 · p. 11