SIDInspector: A Mapping-First Diagnostic Resource for Semantic-ID Tokenizers

Problem

• SID mappings are reused as generator address spaces, but repositories expose heterogeneous artifacts and omit a common inspection interface.Users must determine whether items have valid codes, how many share full codes, and whether prefixes align with behavior or compress the tail.Source: §1 · p. 1
• Downstream ranking metrics can conceal underused codebooks, repeated full codes, weak prefixes, tail compression, and large prefix fan-out.These are direct properties of reusable tokenizer artifacts rather than only outcomes of trained recommenders.Source: §1 · pp. 1–2

Contribution

• SIDInspector profiles capacity use, aliasing, neighborhood alignment, head-tail allocation, and structural cost before a full generator is retrained.The resource is designed for inspection before or alongside downstream recommendation evaluation.Source: §1 · p. 2
• Its resource combines a validated adapter contract, D1–D5 mapping probes, D6 churn support, and worked examples spanning GRID/RQ-KMeans, ReSID/GAOQ, LETTER, and LC-Rec.Controls and mechanism probes distinguish addressability from behavioral neighborhood alignment.Source: §1 · p. 2

Artifact boundary

• SIDInspector treats a tokenizer export as an inspectable resource artifact rather than an opaque component of a trained recommender.The minimum input is an item-to-code table, while metadata, interactions, refresh pairs, and generator outputs are optional extensions.Source: §2 · p. 2

Adapter layer

• Adapters export one normalized table with method, dataset, provenance, and one discrete token per SID level; checkpoint-only releases require upstream item-level export.The metric code remains unchanged after integration and preflight.Source: §2 · p. 2

Normalized contract

• The contract requires stable item keys and consistent SID depth, with optional tables enabling semantic, collaborative, head-tail, churn, and generation-trace analyses.Variable-length tokenizers can use padded or masked levels, but without traces their serving interpretation remains limited to structural D5 evidence.Source: §2 · p. 2

Validation gate

• Validation checks item counts, keys, discrete tokens, depth consistency, optional joins, and provenance before metrics run.This prevents checkpoints, mappings, codebooks, features, and partial releases from entering one comparison under the same label.Source: §2 · p. 2

Probe engine and reports

• D1–D5 reports cover utilization, aliasing, neighborhood alignment, popularity allocation, and structural cost, while D6 supports paired refresh mappings and D7 remains a trace-input hook.Reports preserve artifact labels, dataset, coverage, and provenance and are intended to guide follow-up inspection rather than impose universal thresholds.Source: §2 · p. 2

Scope

• SIDInspector fixes the inspected object at the item-to-code address space consumed by generative recommenders before a reranker or generator is trained.This distinguishes its artifact boundary from broader recommender evaluation resources.Source: §2 · p. 2

Interpretation

• Table 1 maps each D1–D5 signal to a follow-up inspection action rather than a universal pass/fail rule.The diagnostic scale is intended to support calibrated investigation, not a single acceptance threshold.Source: Table 1 · p. 3

Mapping probes

• D2 measures the fraction of items in non-singleton full-code or prefix alias sets, while warning that mapping aliasing alone does not establish causal harm.A bounded interaction-qualified mechanism probe provides an additional check because duplicate assignments are not uniformly harmful.Source: §3 · p. 3

Neighborhood alignment

• D3 tests whether train-only co-occurrence neighbors share an item's prefix, reporting weighted edge and mean item scores relative to same-dataset controls.It diagnoses candidate exposure rather than ranking quality, and metadata purity is only auxiliary context.Source: §3 · p. 3

Allocation and structural cost

• D4 compares head, middle, and tail allocation of full-code capacity and prefix structure, while D5 reports length, fan-out, duplicates, and trie-like expansion pressure.These are structural signals rather than latency measurements; serving cost requires implementation-specific evaluation.Source: §3 · p. 3

Same-item profiles

• Table 2 compares artifact profiles on 23,742 Musical items, with bold rows forming the primary contrast and italic rows serving as controls.Its columns cover D2 full-code aliasing, D3 level-1 neighborhood alignment, D4 tail uniqueness, and D5 active-prefix counts by depth.Source: Table 2 · p. 3

Optional probes

• D6 computes mapping changes across tokenizer refreshes, whereas D7 is designed for invalid paths, next-token entropy, and duplicate generated candidates but requires generator outputs or beam traces.V0 therefore implements D6 as an optional refresh-pair extension and supplies only a D7 input hook.Source: §3 · p. 3

Same-item protocol

• 0.000 full-code aliasing and higher D3-L1 recovery make ReSID/GAOQ structurally distinct from the GRID-style rows, while category control provides the alignment reference.The same-item Musical protocol holds the 23,742-item universe fixed; zero aliasing alone does not establish recommendation quality.Source: §4 · pp. 3–4
• 0.977 to 0.779 D2 aliasing under the larger GRID-style budget shows capacity reduces collisions, but 9,874 unique codes still cover only 23,742 items.The comparison remains an artifact-profile result, not a method-quality claim.Source: §4 · pp. 3–4

Findings

• 0.055–0.080 D3 remains below the category-prefix control’s 0.447, demonstrating that lower aliasing and behavioral prefix alignment are complementary properties.The category row is a deterministic scale control rather than a learned tokenizer competitor.Source: §4 · p. 4

Coverage and portability

• Released LETTER and LC-Rec artifacts both pass validation on 9,922 items, exposing 9,897 unique full SIDs with D3-L1 values of 0.109 and 0.052.These rows test ingestion portability on a different item universe rather than same-item method ranking.Source: §4 · p. 4

Dataset effects

• 0.968 category-control D3-L1 on All_Beauty versus 0.081–0.090 for GRID feature-text rows shows that D3 scale depends strongly on available category signal.The result is a dataset-dependent calibration, not a universal threshold.Source: §4 · p. 4

Candidate exposure

• 0.976 Musical and 1.000 All_Beauty Spearman correlations connect D3 with train-only prefix candidate recall, while fixed-reranker correlations remain diagnostic exposure checks.The probes do not reject tokenizers or predict trained-generator quality.Source: §4 · p. 4

Availability

• The MIT-licensed v1.0 release includes an adapter template, examples, tests, documentation, and a clean-checkout verifier for rebuilding the reported tables.Snapshots reproduce tables rather than upstream tokenizer training or omitted caches.Source: §5 · pp. 4–5

Uses

• Researchers can flag missing mappings, inconsistent depth, alias groups, weak collaborative-prefix recovery, and tail compression before generator training.Method authors can emit normalized SID tables and reuse the validator and D1–D5 engine; the toolkit supplies next actions rather than universal thresholds.Source: §5 · p. 4

Scope

• The four tokenizer lines have explicit provenance, while All_Beauty, Sports, MovieLens, and DACT check dataset effects, schema portability, and optional churn rather than method ranking.Controls and mechanism rows are not additional named-method coverage.Source: §5 · p. 4

Limitations

• D2, D3, and fixed-reranker probes remain artifact-level or candidate-exposure diagnostics; D5 is not serving latency, D7 remains an input hook, and broader decoding evaluation is future work.The paper explicitly limits conclusions about trained-generator quality and downstream interventions.Source: §5 · p. 4

• Generative-AI tools assisted with language editing and code debugging, while author-written scripts produced and verified all experimental results.Source: GenAI Usage Disclosure · p. 5