Residual Trajectory Distillation for Generative Retrieval

• Hard SID targets omit residual preferences that can differ even when items share the same selected code, leaving information from indexing unused in retrieval training.Residuals also influence later quantization decisions, creating a gap between identifier construction and retrieval supervision.Source: §1 · pp. 1–2
• Residual preferences can conflict with retrieval addresses because nearby codewords may identify another item or an invalid path, and collision resolution can change the stored SID.Therefore, residual supervision must preserve information from quantization while remaining consistent with the stored address.Source: §1 · p. 2
• ResTD uses the frozen RQ indexer as a process teacher to transfer current and future residual-induced preferences into decoder states without changing the identifier space or inference procedure.Earlier states receive preferences from subsequent quantization levels before the corresponding SID suffix is generated.Source: §1 · pp. 2–3
• ResTD consistently outperforms strong baselines on ESCI retrieval in English, Spanish, and Japanese, with controlled comparisons and probes supporting residual-derived supervision.The paper also reports consistent improvements on multiple generative recommendation datasets.Source: §1 · pp. 2–3

• Generative retrieval research develops semantic identifiers and ranking-aware or autoregressive methods, while ResTD derives supervision from residual quantization of a fixed index.Examples include TIGER’s residual-quantized Semantic IDs and ranking-aware LTRGR.Source: §2 · p. 3
• Knowledge distillation and quantized-generation work use soft targets or intermediate representations; ResTD instead distills residual preferences reconstructed along stored SID paths from the frozen indexer.Related approaches include residual-distance targets in RQ-Transformer and item-code alignment in ETEGRec.Source: §2 · p. 3
• Future-supervision methods train earlier sequence states using later information, and ResTD uses residual codebook distributions as structured targets for subsequent quantization levels.Related examples include Twin Networks, ProphetNet, and multi-token prediction.Source: §2 · p. 3

• RQ represents an item through successive vector quantization over level-specific codebooks, selecting a codeword at each level and updating the residual for the next.After L levels, the indexer produces both a Semantic ID and a residual trajectory of intermediate states.Source: §3 · p. 3
• A generative retriever predicts the target SID autoregressively from a query using the standard sequence-generation objective under teacher forcing.Decoder states are associated with predicting SID tokens; auxiliary supervision is applied only to states that directly predict them.Source: §3 · pp. 3–4
• ResTD’s overview aligns decoder states with residual-derived teacher distributions supplied along stored SID paths, using separate heads for current and future targets.The framework description identifies the frozen RQ indexer as the source of these targets.Source: Figure 2 · p. 4
• Identical hard SID targets can conceal different residual distances and margins to alternative codewords within the same quantization cell.The distinction between the selected code and its underlying residual geometry motivates the added supervision.Source: §3 · p. 4

4.1 RESIDUAL BEHAVIORAL DISTILLATION

• RBD matches residual-induced codeword preferences to projected decoder-state distributions in the frozen codebook space, while the original SID objective continues to supervise retrieval.Only valid SID positions contribute to the distillation loss.Source: §4.1 · pp. 4–5

4.2 MULTI-HORIZON RESIDUAL TRAJECTORY DISTILLATION

• Early decoder states receive soft preferences for later codebooks before suffix tokens enter context; valid horizons shorten toward the SID end and distant horizons are discounted.The horizon controls the range and relative weighting of future supervision; H = 1 recovers current-level RBD.Source: §4.2 · p. 5

4.3 COLLISION-AWARE TEACHER CONSTRUCTION

• Collision-aware construction rebuilds residuals along the stored SID, then minimally mixes toward its code to enforce a target margin while preserving alternatives’ pairwise ratios.The correction is the minimum adaptive perturbation, subject to a floor, needed to make the stored code preferred by the teacher.Source: §4.3 · pp. 5–6

4.4 TRAINING AND INFERENCE

• Training adds residual trajectory distillation to the unchanged retrieval objective with a bounded, warmed-up contribution; inference removes teachers and projections and retains the original index and search procedure.The RQ indexer and codebooks remain fixed during training; retrieval uses the original SID-generation distribution and trie-constrained autoregressive search.Source: §4.4 · p. 6

5.1 EXPERIMENTAL SETUP

• The evaluation covers three ESCI locales and compares ResTD with sparse, dense, and generative retrievers under published protocols and matched CaLIR continuations.Controlled runs use three paired seeds, a shared 1,200-update budget, and trie-constrained beam search.Source: §5.1 · pp. 6–7

5.2 EXPERIMENTAL RESULTS AND DISCUSSION

• ResTD achieves the best results on every reported metric across ESCI-US, ESCI-ES, and ESCI-JP, outperforming sparse, dense, and generative retrieval baselines.The gains include stronger early ranking and increased Recall@100 in every dataset.Source: Table 1 · p. 7, §5.2 · p. 7
• Under matched training, current-level RBD improves every reported metric over Base-FT, and extending ResTD supervision to H = 2 and H = 4 yields progressive gains.ResTD also surpasses future hard-SID supervision; collision-aware correction further improves Recall@100 and NDCG@10 in every locale.Source: §5.2 · pp. 7–8, Table 2 · p. 8
• Code-conditioned residual distributions outperform future hard-SID and codebook-only targets, while item-specific residual preferences improve every reported metric over shape-matched teachers from other items.Preserving codeword-to-probability correspondence also improves retrieval over non-target permutation.Source: §5.2 · p. 8, Table 3 · p. 8
• ResTD reduces probe KL by 5.3–13.8% on every future state–target pair versus current-level RBD, while current-target reductions remain 0.1–0.3%.The largest reduction in each locale occurs when the first decoder state is probed against the fourth-level teacher distribution.Source: §5.2 · p. 8, Figure 3 · p. 9
• Retrieval gains generally increase with teacher disagreement on ESCI-US and ESCI-JP, but ESCI-ES NDCG@10 gains peak in the intermediate-disagreement group.After adjustment, associations remain positive on US and JP, while ES confidence intervals span zero.Source: §5.2 · pp. 8–9, Figure 4 · p. 9
• With TIGER as its backbone, ResTD achieves the best performance across all reported dataset–metric settings for Beauty, Instruments, and Yelp recommendation.Gains are strongest on Beauty and Yelp and positive on both Instruments metrics.Source: Table 4 · p. 9, §5.2 · p. 9
• In the qualitative examples, ResTD retains the target identifier path pruned by the baseline in the Casio case and maintains relevant candidates through later decoding in US and JP examples.The examples show improved target ranking and retrieval of complementary products.Source: Figure 5 · p. 9, §5.2 · p. 9

6 CONCLUSION

• Across three ESCI locales, ResTD outperforms continued training, current-level RBD, and future hard-SID supervision under matched settings, and results extend to recommendation.Controlled comparisons and representation probes support residual-derived targets and improved recovery of future codebook preferences from earlier decoder states.Source: §6 · p. 10
• The authors report reviewing potential ethical implications and identifying no particular issues warranting additional discussion.They state that generative AI was used solely to refine wording and readability, not to formulate questions, design methods or experiments, or analyze results.Source: ETHICS STATEMENT · p. 10, AI USE STATEMENT · p. 10

Appendix

• The paper provides method, experimental-configuration, and evaluation details in the main paper and appendix, with source code and implementation details available for reproduction.The appendix includes additional NQ320K document-retrieval experiments and TIGER-based recommendation experiments.Source: REPRODUCIBILITY STATEMENT · p. 10, A EXPERIMENTAL SETUP AND IMPLEMENTATION · p. 13

A.1 RETRIEVAL DATASETS

• ESCI evaluation uses filtered product sets of 288,372 items for US, 101,957 for ES, and 119,052 for JP, with test sets of 6,014, 1,656, and 1,883 queries, respectively.The benchmark reports Recall@5, @10, and @100 and NDCG@10 and @100.Source: §A.1 · p. 13

A.2 RETRIEVAL IMPLEMENTATION DETAILS

• The collision-resolved SID index has four quantization levels with 256 codewords per level, and residual targets follow the final SID paths.Controlled comparisons use a fixed index and evaluation protocol.Source: §A.2 · pp. 13–14
• All ESCI locales use H = 4, λmax = 0.1, τT = τS = 0.2, and ρ = 0.7, with a collision-correction floor of ϵ = 0.1 and margin δ = 0.001.Each condition has four auxiliary heads, so changing H varies supervised horizons without changing architecture.Source: §A.2 · p. 14
• Each condition trains for 1,200 updates with effective batch size 128, learning rate 10^-5, and cosine decay, using seeds 42, 2027, and 2028.Within each seed, conditions share initialization, data, minibatch order, index, frozen category adapter, schedule, and evaluation protocol; weighted auxiliary loss is capped at 5% of SID loss.Source: §A.2 · p. 14

A.3 EVALUATION PROTOCOL AND UNCERTAINTY

• Fine-tuning uses trie-constrained decoding with beam width 100 and category width three, with the category adapter frozen and shared across objectives.Reported means and confidence intervals use this decoding protocol; Recall and NDCG are averaged over test queries.Source: §A.3 · p. 14
• Table 2 reports means and standard deviations across three seeds; paired differences measure variation across seeds, while query resampling measures uncertainty conditional on fitted models.Matched future-target controls assess multi-horizon supervision.Source: §A.3 · p. 15

A.4 BASELINES AND MATCHED CONTROLS

• Table 1 compares ResTD with sparse, dense, and generative methods, using published protocols for external baselines and reproduced CaLIR scores.ResTD scores are three-seed H = 4 means from Table 2.Source: §A.4 · p. 15

A.4.1 SPARSE RETRIEVAL

• BM25 is the sparse lexical baseline, scoring query–item pairs using term frequency and inverse document frequency with saturation and document-length normalization.It requires no learned representations.Source: §A.4.1 · p. 15

A.4.2 DENSE RETRIEVAL

• Dense baselines include DPR, MPNet, Sentence-T5, multilingual Sentence-mT5, BGE-M3 dense representations, and LaSER.Their described approaches use shared-space similarity, pretrained sentence representations, multilingual embeddings, or reasoning distillation for retrieval.Source: §A.4.2 · p. 15

A.4.3 GENERATIVE RETRIEVAL

• Generative baselines range from arbitrary or semantic identifiers to residual-quantized TIGER and methods incorporating category, relevance, ranking, or intent signals.The described methods include DSInaive, DSIsemantic, TIGER, Hi-Gen, LTRGR, RIPOR, CAT-ID2, MERGE, FORGE, HGRec, and CaLIR.Source: §A.4.3 · pp. 15–16

A.4.4 MATCHED TRAINING CONTROLS

• Matched controls compare Base-FT, current-level RBD, multi-horizon residual or hard-SID targets, shuffled teacher pairings, full-output KD, and non-target output alignment.The controls begin from reproduced CaLIR and are evaluated in Section 5.2 and Appendix C.3.Source: §A.4.4 · p. 16

A.5 GENERATIVE RECOMMENDATION SETUP

• Recommendation experiments use Beauty, Instruments, and Yelp after 5-core filtering, with chronological interaction histories and penultimate and final interactions held out for validation and testing.A frozen RQ-VAE tokenizer creates three semantic codes plus a disambiguation token; a TIGER-style encoder–decoder predicts the next item's SID autoregressively.Source: §A.5 · pp. 16–17

B.1 INFORMATION RETAINED BY RESIDUAL TEACHERS

• Within a quantization cell, residual distances vary despite a fixed winning code, so residual-derived teacher distributions preserve item-specific codeword preferences absent from hard labels.The paper states that these distributions preserve the ordering and margins among competing codes.Source: §B.1 · pp. 17–18

B.2 GRADIENTS OF HARD AND RESIDUAL SUPERVISION

• Residual and hard targets change the student gradient according to the teacher's codeword probabilities, while using the same heads and supervised horizons.For the future hard-SID control, the target-dependent gradient changes while supervision locations remain fixed.Source: §B.2 · p. 18

B.3 NORMALIZED HORIZON WEIGHTS

• With L = 4, H = 4, and ρ = 0.7, normalized horizon weights are (0.5389, 0.2829, 0.1320, 0.0462), whereas H = 1 assigns all weight to the current level.Increasing the horizon changes both supervised targets and their relative weights; realized loss and gradient contributions also depend on KL terms, model Jacobians, and scaling.Source: §B.3 · p. 18

B.4 TRAINING ALGORITHM

• The training algorithm precomputes collision-corrected teacher distributions along stored SID paths, then applies weighted KL supervision while keeping the indexer frozen.Auxiliary heads and teacher distributions are discarded at inference.Source: §B.4 · p. 19

B.5.1 COMPUTATIONAL COMPLEXITY

• Horizon heads and precomputed residual trajectories add training computation and storage, but these additions incur no inference cost.Distance computations can be parallelized over positions, horizons, and codewords.Source: §B.5.1 · p. 19

B.5.2 MEASURED TRAINING OVERHEAD

• Across 27 full-length experiments, H = 4 training takes 1.116×, 1.083×, and 1.094× Base-FT time on US, ES, and JP.On ESCI-US, an update takes 1.119× Base-FT time and uses 31.8 MB more peak allocated memory.Source: §B.5.2 · p. 19, Table 6 · p. 20
• Precomputed residual trajectories require 0.0947 GB total for US, including SIDs, codebooks, and metadata, and 0.0326 and 0.0381 GB for ES and JP trajectories.The same trajectories supervise both H = 1 and H = 4.Source: §B.5.2 · p. 19

C SUPPLEMENTARY EXPERIMENTS AND ANALYSES

• The supplementary experiments examine NQ320K generalization, teacher targets, output-space supervision, representation probes, multi-positive queries, collision correction, and hyperparameter sensitivity.Source: C SUPPLEMENTARY EXPERIMENTS AND ANALYSES · p. 20

C.1 GENERALIZATION TO NQ320K

• ResTD leads TIGER, MERGE, and CaLIR on every reported NQ320K metric across full, seen, and unseen test sets.On unseen documents, Recall@10 and MRR@100 more than double the strongest baseline scores.Source: Table 7 · p. 20, §C.1 · p. 20

C.2.1 WITHIN-CODE VARIATION IN TEACHER DISTRIBUTIONS

• Among 20,000 ESCI-US pairs matched by first-level hard code and teacher-entropy decile, mean JS divergence is 0.1330 nats and the 95th percentile is 0.3088.A selected pair assigned to c71 has strongest alternatives c192 and c217, with JS divergence 0.5322 nats.Source: §C.2.1 · p. 21, Figure 6 · p. 21

C.2.2 ADDITIONAL RETRIEVAL RESULTS

• Control targets vary whether they preserve hard codes, teacher-distribution shape, real-item distributions, and item–teacher pairing; Table 9 extends their retrieval comparison to five metrics.Codebook-only targets use stored codes and frozen codebooks without item residuals or corpus residual statistics.Source: §C.2.2 · p. 21, Table 9 · p. 22, §C.2.3 · p. 22

C.2.3 CONTROL CONSTRUCTION

• Teacher controls are fixed per valid item–level pair for current and future state–horizon targets, while the future-hard-SID control replaces only future targets.A separate non-target output-alignment objective renormalizes probabilities over alternative codes and leaves the gold-SID objective unchanged.Source: §C.2.3 · p. 22, §C.3.2 · p. 25

C.2.4 MATCHING QUALITY

• Shape-matched teacher swaps achieve 99.76–99.91% different-item coverage, with median entropy mismatch of 0.008–0.012 nats and median Shape L1 of 0.018–0.026.Similar profiles can still have different codeword probabilities.Source: §C.2.4 · p. 24, Table 11 · p. 24

C.3.1 FULL-DISTRIBUTION OUTPUT DISTILLATION

• Current-level RBD adds 0.10, 0.06, and 0.09 percentage points over full-output KD in Recall@100 on US, ES, and JP, respectively.Its NDCG@10 gains over full-output KD are 0.09, 0.07, and 0.11 points.Source: §C.3.1 · p. 24, Table 12 · p. 24

C.4.1 CURRENT-TARGET DIAGNOSTIC

• Current-level RBD reduces held-out probe KL by 0.06–1.84% relative to Base-FT across three locales and four SID levels.The future-target diagnostic compares held-out KL at current and later SID positions using the collision-corrected training teacher.Source: §C.4.1 · p. 25, Figure 7 · p. 25, §C.4.2 · p. 26

C.5.1 ITEM-CONDITIONED TEACHER TARGETS

• For a shared decoding context, the mean teacher distribution minimizes expected forward-KL loss, while teacher disagreement contributes a student-independent generalized JS term.Multi-positive queries have mean JS divergences of 0.389, 0.557, and 0.543 nats on US, ES, and JP.Source: §C.5.1 · p. 26, §C.5.2 · p. 26

C.5.4 ADJUSTED ASSOCIATIONS

• Each 0.1-nat increase in teacher JS is associated with adjusted NDCG@10 gains over H = 1 of 0.17 points on US, 0.06 on ES, and 0.62 on JP.The ES confidence interval includes zero (−0.19–0.31); the US and JP intervals are 0.08–0.25 and 0.39–0.85.Source: §C.5.3 · p. 26, Table 14 · p. 27, §C.5.4 · p. 28

C.6.1 FREQUENCY AND MAGNITUDE OF TEACHER–SID CONFLICTS

• At the fourth quantization level, teacher–SID conflict rates reach 10.1%, 12.9%, and 11.7% on US, ES, and JP, while correction with ϵ = 0.10 improves Recall@100 over raw teachers.The Recall@100 gains are 0.38, 0.34, and 0.43 percentage points, respectively; increasing the floor to 0.20 reduces scores by 0.07–0.11 points versus 0.10.Source: §C.6.1 · p. 28, Table 15 · p. 28, §C.6.2 · p. 29

C.7 HYPERPARAMETER SENSITIVITY

• Increasing λmax does not improve scores monotonically: 0.05 gives the highest Recall@10 on US and JP, while ES peaks at the default 0.1.On JP, NDCG@10 instead peaks at 0.15, and the effective auxiliary contribution is capped at 5% of SID loss.Source: §C.7 · p. 29, Figure 8 · p. 30
• Jointly varied τT = τS values peak at 0.3 on US, 0.2 on ES, and 0.05 on JP for both metrics in the tested grid.Because teacher and student temperatures vary together, this sweep does not isolate teacher softness from student optimization.Source: §C.7 · p. 29, Figure 8 · p. 30
• The largest plotted scores occur at ρ = 0.9 on US, 0.5 on ES, and 0.3 on JP; every tested discount retains all four supervision offsets.The comparison changes the allocation of trajectory supervision while keeping its horizon range fixed.Source: §C.7 · p. 29, Figure 8 · p. 30
• The one-at-a-time sweeps do not evaluate interactions among hyperparameters.The shared defaults specify a common configuration across locales.Source: §C.7 · p. 29

D QUALITATIVE CASE STUDY

• In selected examples, ResTD raises a Huawei P30 Lite case from rank 4 to 1 and a gasket-free Zojirushi bottle from rank 25 to 2.The queries require phone-model compatibility and a combination of brand and design requirements, respectively.Source: D QUALITATIVE CASE STUDY · p. 31, Figure 9 · p. 31
• ResTD also promotes judged substitutes and complements, including a brake controller from rank 2 to 1 and a TV stand from rank 10 to 1.A camera lens enters the top ten at rank 6 from rank 13; both models retrieve the complete target SID in all six examples.Source: D QUALITATIVE CASE STUDY · p. 31, Figure 9 · p. 31
• The six selected cases span US, ES, and JP and include Exact, Substitute, and Complement judgments.The examples compare each judged target with CaLIR’s top-ranked product.Source: Figure 9 · p. 31