Source-linked AI summary
The Vocabulary Is Already Here: Adapting Pretrained Interpretable Features for Generative Recommendation
Ziluowen Luo, Ruochen Liu, Jun Yin, Hao Yan, Zhengxuan Hu, Xinze Shi, Chaozhuo Li, Chengqi Zhang, Senzhang Wang
TL;DR
Generative recommendation needs compact item identifiers that preserve distinctions and provide effective supervision, while existing fitted vocabularies incur additional costs and domain-dependent coverage. API adapts pretrained interpretable features into item IDs and preference learning; experiments report competitive recommendation and explainable retrieval.
Problem
Generative recommendation needs compact semantic IDs that preserve item distinctions and provide effective supervision, while fitted vocabularies incur additional costs and depend on fitting domains.
Source: §1 · p. 2Method
API reuses pretrained sparse-autoencoder feature indices as a vocabulary and combines strength-aware PIDs with SPA preference learning and feature-based retrieval explanations.
Source: §1 · p. 2Results
Experiments support pretrained vocabulary reuse as a viable basis for generative recommendation, with competitive recommendation and explainable retrieval.
Source: §1 · p. 2Takeaways & Limitations
API connects compact item representations and user preference learning with interpretable retrieval evidence using a pretrained dictionary adapted for recommendation.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Generative recommendation (GR) predicts the next item through its semantic ID (SID). Existing methods commonly fit SID vocabularies to pretrained language model (PLM) representations of item descriptions, incurring additional training costs and making vocabulary coverage dependent on the fitting domains. We observe that pretrained sparse autoencoders expose PLM features with human-interpretable semantic annotations, allowing feature indices to serve as a reusable SID vocabulary. However, reusing this vocabulary requires preserving item distinctions under compact token budgets and learning to favor target items over candidates with shared dominant features. It also requires extending feature interpretability from item content to retrieval decisions. To address these challenges, we propose API, a framework that Adapts the Pretrained Interpretable vocabulary for effective and explainable GR without additional vocabulary learning. Within API, Pretrained Interpretable IDs (PIDs) retain selected feature indices and relative activation strengths for item discrimination. With PIDs as supervision, the Semantic Prediction Adapter (SPA) learns user preferences from interaction histories through joint feature-index and strength prediction. SPA combines both predictions to score candidate items, with item-level supervision encouraging the observed next item to outrank its alternatives. API further explains candidate score differences through feature contributions paired with their pretrained semantic annotations. Extensive experiments show that API achieves competitive performance across backbones and decoding schemes, while preserving item distinctions under compact token budgets and enabling explainable retrieval.
1 Introduction
Generative recommendation depends on compact semantic IDs that preserve item distinctions and provide effective prediction supervision, but learned vocabularies require recommendation-specific fitting. API reuses pretrained interpretable features as a vocabulary and adapts item representations, preference learning, and retrieval explanations.
1 Introduction
- Recommendation tokenization must preserve item distinctions while providing effective token-level supervision, making compact SID construction a central design problem.The item-to-SID mapping affects both item representation and prediction supervision.Source: §1 · p. 2
- Existing SID vocabularies are commonly fitted to PLM representations, adding optimization costs and making semantic coverage depend on fitting data.Fitted tokens also need not have readily interpretable semantics.Source: §1 · p. 2
- Pretrained sparse autoencoder feature indices can serve as a reusable SID vocabulary, with activations encoding strengths and annotations making feature semantics inspectable.The vocabulary can be reused without recommendation-specific vocabulary learning.Source: §1 · p. 2
- Compact PIDs must retain item distinctions beyond shared dominant features, while prediction must make the target outrank candidates sharing those features.Indices alone may discard distinctions carried by activation strengths and less active features.Source: §1 · p. 2
- Retrieval explanation requires tracing feature contributions to items to explain why the model favors one candidate over another.Source: §1 · p. 2
- API combines PID construction with SPA preference learning and explains retrieval through feature-level contributions linked to feature semantics.PIDs retain selected feature indices and activation evidence; SPA uses feature identities and relative strengths, with preference alignment against competing items.Source: §1 · p. 2
2 Related Works
Prior generative recommendation methods construct item identifiers through learned or fitted tokenization, while SAE research exposes interpretable features in language-model representations. This work adapts a pretrained SAE dictionary as a shared vocabulary for generative retrieval.
2 Related Works
- Structured and text-based recommendation methods construct item identifiers through approaches including residual quantization, tree indexing, collaborative semantics, and cross-domain shared tokenization.Despite these advances, vocabulary learning still relies on recommendation-specific data, and its semantic organization remains limited by training domains.Source: §2 · p. 2
- Sparse autoencoders expose interpretable structure in language-model activations, and API reuses a pretrained SAE feature dictionary as a shared vocabulary for generative recommendation.Unlike SAERec, which uses SAE-derived intent priors, this work connects feature-level concepts to generative retrieval.Source: §2 · p. 3
3 Preliminaries
Generative recommendation predicts a user's next item by generating tokens from interaction history and mapping the sequence to an item. Item tokenization supplies compact identifiers, while pretrained sparse autoencoders provide a feature vocabulary with activation strengths and semantic annotations.
3 Preliminaries
- Generative recommendation predicts the subsequent item from a user's sequential interaction history by generating a discrete token sequence and mapping it to items.Autoregressive decoding generates tokens sequentially; parallel decoding predicts multiple tokens in one step.Source: §3 · p. 3
- Item tokenization converts PLM representations of item descriptions into compact semantic IDs drawn from a shared vocabulary, determining retained information and token-level supervision.A SID is a sequence of K discrete tokens.Source: §3 · p. 3
- A pretrained sparse autoencoder maps PLM representations to sparse features whose indices identify features, activations indicate their strengths, and annotations describe their semantics.These feature indices can serve as a reusable token vocabulary accompanied by activation strengths and readable annotations.Source: §3 · p. 3
4 Methodology
API builds compact item identifiers from pretrained interpretable features, then uses a Semantic Prediction Adapter to predict features and strengths from interaction histories. Item-level supervision guides candidate ranking, and feature contributions explain retrieval decisions.
4.1 Pretrained Interpretable IDs Construction
- PIDs select strongly activated, less prevalent features and retain relative activation strengths, helping distinguish items that share the same feature-index sequence.The separability bound relates collisions to reconstruction error and variation in summed strengths; when Δ > 2ε + δ, strengths can distinguish items with identical token sequences.Source: §4.1 · pp. 3–4
- PIDs combine selected feature indices, their relative strengths, and corresponding pretrained semantic annotations.Feature selection considers the M strongest positive activations, favors strong but less prevalent features, and orders selected indices by activation strength.Source: §4.1 · pp. 3–4
4.2 SPA: PID Prediction
- SPA encodes historical-item feature–strength pairs and predicts the next item’s feature-index sequence and relative strengths from the interaction history.The history representation includes a candidate pool of features, not only the selected PID indices.Source: §4.2 · p. 4
4.3 SPA: Item-Level Supervision
- SPA scores candidates by balancing feature-sequence likelihood with agreement between predicted and candidate relative strengths.Item-level preference alignment encourages the observed next item to outrank competing candidates; the stated bound guarantees ranking above a same-sequence competitor when strength-prediction error is below Δq/4.Source: §4.3 · pp. 4–5
4.4 Retrieval Explanation
- API decomposes candidate score differences into interpretable feature contributions paired with pretrained semantic annotations.This extends explanations from item representations to why one candidate is favored over another.Source: §4.4 · p. 5
5 Experiments
Across Beauty, Sports, and Toys, the experiments assess recommendation performance, PID discrimination and content preservation, supervision, vocabulary reuse, and retrieval explanation. API achieves the best reported result in nine of twelve dataset–metric comparisons.
5.1 Experimental Setup
- Experiments use Beauty, Sports, and Toys Amazon subsets, evaluating full-catalog Recall@5/10 and NDCG@5/10 alongside PID collision, content overlap, and steering exposure.The comparisons include T5- and GPT-2-style backbones and autoregressive and parallel decoding.Source: §5.1 · pp. 5–6
5.2 Recommendation Performance (RQ1)
- API achieves the best reported result in nine of twelve dataset–metric comparisons, including all four metrics on Toys.It ranks second on Beauty Recall@5/NDCG@5 and Sports Recall@10, and ties on Beauty NDCG@5.Source: Table 2 · p. 6, §5.2 · p. 7
5.3 PID Quality (RQ2)
- At K=4, PID reduces mean collision rate across three datasets from 80.76% under native selection to 17.09%.Random selection has fewer collisions but weaker content alignment; permuting feature–activation profiles leaves collisions unchanged while reducing Overlap@20 below 0.2%. Activation-weighted neighborhoods also exceed index-only neighborhoods in PLM-based Overlap@20 across all benchmarks.Source: Table 3 · p. 7, §5.3 · p. 7, Table 4 · p. 7
5.5 Vocabulary Reuse (RQ4)
- Across target catalogs, PID reuses the frozen P-SAE vocabulary without refitting, leading on Toys and Sports under both source configurations while R-SAE leads on Beauty.PID also has comparatively low collision rates and competitive Recall@5 across targets; the figure reports collision rates and Recall@5 for each source–target pair.Source: §5.5 · p. 7, Figure 3 · p. 8
5.6 Retrieval Explanation (RQ5)
- For Sports feature f4260, 77 of 78 training items with the feature in their K=4 PIDs match the hunting-and-shooting lexical rule, a 7.91× lift over catalog prevalence.The feature’s external descriptions refer to ammunition and weapons, and the highest-activation titles describe training ammunition.Source: Figure 4 · p. 8, §5.6 · p. 9
5.4 SPA Supervision (RQ3)
- Preference alignment raises Recall@10 by 13.3%, 45.9%, and 9.4% on Beauty, Sports, and Toys; strength modeling further improves NDCG@10 when combined with alignment.Strength prediction and scoring alone lowers NDCG@10 on all three datasets, while full SPA achieves the highest Recall@10 and NDCG@10 throughout the ablation.Source: §5.4 · p. 7, Table 5 · p. 8
5.5 Vocabulary Reuse (RQ4)
- Across three projected tokenization tasks, cumulative time is 3.78× for API, 407.69× for RVQ, and 674.99× for UTGRec relative to one GPT-2 encoding task.The estimate counts UTGRec pretraining once and treats the pretrained PLM and P-SAE as available resources; API still requires per-task feature extraction and encoding.Source: Figure 6 · p. 8, §5.5 · p. 8
5.6 Retrieval Explanation (RQ5)
- A +0.50 shift in f4260’s standardized log-activation increases target-item exposure@10 by 1.365 percentage points relative to matched control edits on 332 eligible test users.Candidate score decomposition also attributes a +0.2897 LEGO-versus-Bruder score margin to +0.2993 from token likelihoods and −0.0096 from strength agreement.Source: Figure 6 · p. 8, §5.6 · p. 9
6 Conclusion
API adapts a pretrained interpretable dictionary into a shared recommendation vocabulary to address catalog-dependent vocabulary learning. It connects compact item representations and preference learning with interpretable retrieval evidence.
6 Conclusion
- API adapts a pretrained interpretable dictionary into a shared vocabulary to address catalog-dependent vocabulary learning.Source: §6 · p. 9
- API connects compact item representations and user preference learning with interpretable retrieval evidence.Source: §6 · p. 9
- The authors identify more general-purpose interpretable vocabularies and fully realizing their recommendation potential as future directions.Source: §6 · p. 9
A.1 Data and Recommendation Evaluation
The appendix specifies the data and evaluation protocol, pretrained resources, PID construction, and history encoding used for recommendation experiments.
A.1 Data and Recommendation Evaluation
- Evaluation retains the supplied Amazon splits and item orderings, with one validation and one test target per user.Validation uses up to 20 preceding training interactions; test uses up to 20 preceding interactions, including the validation item.Source: §A.1 · p. 10
- Recall@k and NDCG@k are averaged over users at k ∈ {5, 10}, while evaluation ranks item identities without sampled negatives or seen-item filtering.PR scores the full catalog exactly; AR uses beam search over the catalog trie with width 50, and checkpoints are selected by full-validation Recall@5.Source: §A.1 · p. 10
- Table 6 reports dataset statistics after preprocessing.Source: Table 6 · p. 10
A.2 Pretrained Resources and PID Construction
- Item features come from GPT-2-small representations encoded by Bloom’s SAE, with title, brand, category, and price text truncated to 128 tokens.Token-level activations exclude special and padding tokens, and recommendation backbones are randomly initialized.Source: §A.2 · p. 10
- PID construction starts from each item’s top 256 positive features and uses training-item prevalence to adjust feature priority.SPA orders selected features by activation, while direct PID-PR uses training-frequency ordering; a nonsemantic suffix disambiguates shared supports.Source: §A.2 · p. 10
- History encoding standardizes log activations and uses learned gain and shift embeddings to modulate feature embeddings before GELU.Four anchors incorporate projected mean top-32 context and position embeddings, a fifth encodes the suffix, and layer normalization yields five history tokens per item.Source: §A.2 · p. 10
A.3 Backbones, Training, and Baselines
The appendix documents backbone configurations, prediction architectures, training and supervision settings, and the tokenizer and recommendation baselines used for comparisons.
A.3 Backbones, Training, and Baselines
- Table 7 lists recommendation backbone configurations and treats GPT-2 AR and PR as separate capacity cohorts.Source: Table 7 · p. 10
- T5 uses relative positions and a three-layer linear feed-forward network, while GPT-2 uses learned positions with capacities differing by decoding scheme.GPT-2 uses 512 positions for PR and history-plus-target capacity for AR.Source: §A.3 · p. 10
- PR baselines flatten SIDs and predict code positions in parallel, whereas AR uses shifted teacher forcing and candidate-specific prefixes.SPA predicts all roles in one pass without target-token inputs, and its history-only strength prediction is shared across candidates.Source: §A.3 · p. 11
- Recommendation training uses seed 45, gradient accumulation over 16 microbatches, and early stopping within ten epochs or 320,000 optimizer steps.Full validation occurs every 1,600 microbatches; these training limits do not apply to tokenizer fitting.Source: §A.3 · p. 11
- PR preference loss is positive-item cross-entropy over catalog scores, but AR trains with one positive and four distinct uniform negatives rather than exact full-catalog cross-entropy.Full PR-SPA uses (λ, β) = (1, 8); T5-SPA-AR uses (0.25, 2) on Beauty and (1, 8) on Sports and Toys.Source: §A.3 · p. 11
- Tokenizer baselines include OPQ, RVQ, R-KMeans, RQ-VAE, HiD-VAE, native P-SAE, and R-SAE, with residual-tokenizer recipes described separately.The appendix also distinguishes random recommendation codes from random-positive-feature selection and identifies source-transfer recipes as a separate diagnostic cohort.Source: §A.3 · p. 11
- Recommendation-framework comparisons include LC-Rec, UTGRec, Ghost, DiGR, GRU4Rec, and BERT4Rec alongside TIGER-style AR and RPG-style PR.Source: §A.3 · p. 11
B.1 Proof of Proposition 4.1
The proof bounds how items sharing a token sequence can collide under PIDs by relating reconstruction error and activation sums to their strength profiles.
B.1 Proof of Proposition 4.1
- The proof assumes selected-feature reconstructions are within ε of each item’s mean PLM representation and that their activation sums lie in an interval of width at most δ.The activation sums are defined from nonnegative activations with positive totals.Source: §B.1 · p. 11
- For items sharing a PID, equality of their normalized strength profiles bounds their representation distance by reconstruction error and the difference in activation sums.The proof uses the selected-feature reconstruction and the triangle inequality to obtain the bound.Source: §B.1 · p. 11
- When δ < Δ − 2ε, every item in the shared-token group has a distinct PID.The proof establishes a minimum activation-sum gap of Δ − 2ε between distinct items in the group, then concludes that the maximum PID-sharing group has size one.Source: §B.1 · pp. 11–12
B.2 Proof of Proposition 4.2
The proof establishes that SPA ranks the target above a competing item when their feature-index sequences match and the strength-prediction error is sufficiently small. The vocabulary-quality analysis defines collision ambiguity and describes how reuse and content alignment are evaluated.
B.2 Proof of Proposition 4.2
- The proof compares SPA’s predicted relative strengths with the candidates’ PID strength vectors, using a bound on the difference between their strength penalties.It uses the Cauchy–Schwarz inequality to obtain the bound.Source: §B.2 · p. 12
- When two candidates share feature indices, the proof bounds their score difference using strength agreement and shows the target ranks higher if εᵤ < Δq/4.The ranking conclusion also assumes β > 0.Source: §B.2 · p. 12
C.1 Vocabulary Quality
- Collision rate measures the fraction of catalog items sharing a suffix-free key; it captures support ambiguity, not equality of complete PIDs.SAE keys are unordered positive-feature supports, while quantizer keys retain stage identity; strengths are excluded.Source: §C.1 · p. 12
- Content alignment compares each representation’s top-20 neighborhoods with title TF–IDF or PLM-residual reference neighborhoods across fixed domain queries.Index-only neighborhoods use support Jaccard; activation-aware neighborhoods use cosine similarity of sparse, decoder-rescaled activations.Source: §C.1 · p. 12
- Reuse diagnostics freeze source-fitted tokenizers for target-catalog collision tests, while PID keeps the frozen P-SAE dictionary and estimates IDF from target training contents.Comparisons match token counts, and downstream recommenders train on target interactions.Source: §C.1 · p. 12
C.2 Semantic Annotations and Steering
The section evaluates semantic feature annotations and tests whether changing history activations shifts retrieval exposure. A controlled Sports result reports a positive target-minus-control effect while the selected dose retains at least 98% Recall@10.
C.2 Semantic Annotations and Steering
- Feature descriptions come from Neuronpedia and are not model inputs; title agreement measures lexical matches among feature-containing training items, and lift normalizes that rate by catalog match rate.For f4260, the specified whole-word matches include hunting and firearm-related terms.Source: §C.2 · p. 12
- The steering test edits six prespecified features per domain in history activations, keeping supports, anchors, suffixes, and candidate PIDs fixed.Three matched control features and per-query scaling account for history statistics and post-clipping perturbation norms.Source: §C.2 · p. 12
- +1.365 percentage points was the Sports f4260 target-minus-control Exposure@10 effect at dose 0.50, on 332 test users, with interval [0.652, 2.189].Exposure@10 is the fraction of retrieved items whose PID contains the target feature; doses were selected to retain at least 98% Recall@10 on the full split and matched cohort.Source: §C.2 · p. 12
D Additional Results
Additional results examine vocabulary reuse, compact-budget collisions, and semantic examples. PID reduces collisions relative to native P-SAE across target catalogs, while its collision gap narrows as the token budget grows.
Vocabulary reuse
- Across three target catalogs, PID has substantially fewer collisions than native P-SAE with the same frozen dictionary, although RQ-VAE has lower collision rates in this comparison.Figure 7 extends the source-domain comparison to Sports; PID uses the frozen P-SAE dictionary with target-catalog IDF.Source: D Additional Results · p. 12, Figure 7 · p. 13
Collisions under compact budgets
- At K=4, PID reduces collision-affected items from roughly 80% under native selection to 14.64%–21.48% across datasets; the gap narrows at larger budgets.Random-positive selection and RQ-VAE often yield still fewer collisions; collision statistics exclude strengths and disambiguation suffixes.Source: D Additional Results · p. 12, Figure 8 · p. 13
Additional semantic examples
- At K=4, title agreement is 92.4% for Beauty brushes and 99.6% for Toys LEGO, with lifts of 18.31× and 13.77×, respectively.Top-activating titles exhibit the same patterns, providing complementary evidence for interpreting the selected features.Source: D Additional Results · pp. 12–13