HypRQ-VAE: Hyperbolic Item Indexing for Long-Tail-Aware Generative Recommender Systems

Longfeng Wu; Tong Zeng; Giovanni Seni; Zhimin Peng; Bhanu Pratap Singh Rawat; Si Zhang; Yao Zhou; Lecheng Zheng; Bo Ji; Yujun Yan; Dawei Zhou

Source-linked AI summary

HypRQ-VAE: Hyperbolic Item Indexing for Long-Tail-Aware Generative Recommender Systems

Longfeng Wu, Tong Zeng, Giovanni Seni, Zhimin Peng, Bhanu Pratap Singh Rawat, Si Zhang, Yao Zhou, Lecheng Zheng, Bo Ji, Yujun Yan, Dawei Zhou

arXiv:2609.03369v1Sep 3, 2026cs.IR

TL;DR

Generative recommendation must bridge LLM text tokens and discrete item indices while representing long-tail catalogs that Euclidean indexing handles poorly. HypRQ-VAE learns semantic IDs in hyperbolic space through residual quantization and semantic-collaborative alignment, improving recommendation performance, particularly for tail items.

Problem
LLM recommendation uses textual tokens while recommender systems use discrete item IDs, and Euclidean item indexing struggles with long-tailed catalogs.
Method
HypRQ-VAE generates semantic IDs with hyperbolic residual quantization and uses prompts plus fine-tuning to align textual and collaborative semantics.
Results
HypRQ-VAE consistently outperforms Euclidean-based methods across benchmark datasets, with particularly strong gains on long-tail recommendation.
Takeaways & Limitations
Hyperbolic geometry supports more balanced quantization and more expressive item indexing for generative recommendation.

Abstract

from arXiv · show

Sequential recommender systems model user behavior as item ID sequences, while recent generative methods cast recommendation as a language modeling task using large language models (LLMs). While this paradigm incorporates rich textual semantics, it introduces a fundamental mismatch: LLMs operate on text tokens, whereas recommender systems depend on discrete item indices. This misalignment often leads to hallucinations in generative recommendations. Existing methods attempt to bridge this gap by learning item vocabularies in Euclidean space, but they struggle to model the inherent long-tail distribution of real-world catalogs, where a small number of head items dominate, and a vast number of tail items reflect users' niche preferences. To address this issue, we introduce Hyperbolic Residual-Quantized Variational AutoEncoder (HypRQ-VAE), the first framework to learn item indexing in hyperbolic space. HypRQ-VAE leverages the unique properties of hyperbolic geometry, whose exponential volume expansion naturally accommodates the power law structure of user-item interactions. This allows the model to encode rich textual semantics while preserving the representational fidelity of sparse, long-tail items. Experiments on three benchmark datasets show that HypRQ-VAE significantly improves the performance of recommendation, particularly in recommending tail items. Our analysis attributes these gains to the superior capacity of hyperbolic space to model item hierarchies and sparsity in generative recommendation. Our code and data are available at: https://github.com/wulongfeng/HypRQ-VAE.

I. INTRODUCTION

Generative recommendation must reconcile textual LLM representations with discrete item IDs, while Euclidean indexing struggles with long-tailed catalogs. HypRQ-VAE addresses this mismatch by indexing items hyperbolically and is evaluated for its effectiveness, especially on tail items.

Motivation: LLM-based recommendation faces a vocabulary mismatch between textual tokens and the discrete item IDs used to model user behavior.This mismatch can hinder accurate item generation.
Motivation: Raw item IDs create large, semantically meaningless vocabularies, motivating specialized indexing mechanisms such as RQ-VAE.RQ-VAE recursively quantizes residual vectors to produce hierarchical item codewords.
Long-tail challenge: Long-tailed recommendation data contains dominant head items and numerous tail items representing niche interests, while Euclidean methods disproportionately emphasize the head.Item semantics also exhibit a long-tailed distribution, with rare items typically more distinctive.
Proposed approach: HypRQ-VAE generates semantic IDs in hyperbolic space to exploit its capacity for hierarchical structures and rare semantic concepts.The framework compares hyperbolic and Euclidean models using H20 and T80 item groups.
Evaluation: Experiments on Instruments, Arts, and MovieLens demonstrate effectiveness against competing indexing strategies, particularly for tail items.The analysis examines how hyperbolic representations contribute to the observed improvements.

II. PRELIMINARIES

The preliminaries describe Riemannian and hyperbolic geometry, emphasizing the Poincaré ball, manifold mappings, and Möbius operations used by HypRQ-VAE. Figure 3 then illustrates how hyperbolic residual quantization produces semantic IDs.

Riemannian Geometry: A Riemannian manifold uses a metric tensor, tangent spaces, geodesics, induced distances, and maps between tangent and manifold spaces.Exponential and logarithmic maps provide the forward and reverse transitions, while parallel transport preserves geometric structure.
Hyperbolic Space: The Poincaré ball is a bounded, negatively curved manifold suited to representing hierarchical and relational data.Its curvature is constant and negative.
HypRQ-VAE Encoding: HypRQ-VAE encodes an item embedding into a latent vector, maps it into hyperbolic space, and initializes residual quantization with the mapped representation.The process then selects nearest code vectors across multiple quantization levels.
HypRQ-VAE Encoding: Sequential hyperbolic quantization updates each residual through Möbius subtraction and outputs the selected code indices as the item’s semantic code.The illustrated code sequence is (1, 3, 2, 9).
Hyperbolic Operations: Möbius subtraction is defined as p ⊖κ q = p ⊕κ (−q), while exponential and logarithmic maps connect tangent and hyperbolic spaces.These operations support the geometric transformations used in the model.

III. METHODOLOGY

HypRQ-VAE uses hyperbolic residual quantization to generate semantic IDs and a prompt-based generative recommender to align textual and collaborative semantics. The methodology is organized into indexing and recommendation stages for long-tail-aware generation.

Overview of the Approach: The framework has two stages: hyperbolic semantic-ID generation and a generative recommender using those structured identifiers.The indexing process is placed on a hyperbolic manifold to address long-tail representation challenges.
Overview of the Approach: The method addresses the mismatch between LLM textual semantics and recommender collaborative semantics through a two-stage semantic integration strategy.This strategy connects item representation with the generative recommendation process.
Generation of Semantic IDs: Hyperbolic RQ-VAE compresses each item’s textual embedding into several learned discrete IDs using exponential and logarithmic maps.The resulting indices capture textual similarity while providing unique representations for specific items.
Generative Recommender: Prompt instructions align textual and collaborative semantics, while fine-tuning on semantic-ID sequences incorporates collaborative knowledge into LLM generation.The resulting process is used to make recommendations.

B. Generation of Semantic IDs

HypRQ-VAE generates compact semantic IDs by recursively quantizing item embeddings in hyperbolic space. This design combines hierarchical indexing with hyperbolic representations intended to better cover diverse and long-tail items.

Hyperbolic Residual Quantization: HypRQ-VAE maps item embeddings into the Poincaré ball and recursively quantizes residuals using learnable hyperbolic codebooks.At each level, it selects the closest code vector and computes the next residual through Möbius subtraction.
Hyperbolic Residual Quantization: The final hyperbolic representation aggregates selected code vectors, maps back to Euclidean space, and reconstructs the original item embedding.The decoder receives the logarithmic-map output as the reconstructed representation input.
Training Objective: The training objective combines reconstruction loss with quantization loss to optimize item reconstruction and codebook embeddings.The quantization loss uses stop-gradient operations and weighting coefficients for the objective components.
Semantic IDs: RQ-VAE produces hierarchical semantic IDs from coarse-to-fine codewords, using smaller codebooks while retaining substantial representational capacity.An example index sequence is formatted as a tokenized semantic ID such as <a_1><b_3><c_2><d_9>.
Collision Handling: A proximity-based cascading reassignment strategy resolves collisions by assigning competing items unique available tokens, moving upward across codebook levels when necessary.Items with smaller hyperbolic distances receive contested tokens first.

C. Generative Recommender with Semantic IDs

The recommender converts users’ chronological item histories into flattened semantic-ID token streams and trains a sequence-to-sequence model to generate the next item ID autoregressively.

Input Representation: A HypRQ-VAE encodes each historical item into a discrete l-length Semantic ID, which is flattened into a chronologically ordered input token stream.The flattened IDs are embedded within a prompt template for the generative recommender.
Next-Item Generation: The model autoregressively generates the next item’s semantic-ID sequence from the user’s interaction history.Training targets the sequence (c_n+1,1, ···, c_n+1,l).
Next-Item Generation: During inference, each next token is selected from the semantic-ID vocabulary according to its conditional probability given prior generated tokens and the input history.The decoding rule maximizes Pθ(v|Ŷ<t, X) over valid vocabulary tokens.

IV. EXPERIMENTS

The experiments evaluate HypRQ-VAE through benchmark comparisons, ablation studies, and representational analyses focused on robustness and long-tail recommendation.

Evaluation Scope: The evaluation covers datasets, baselines, implementation details, main results, ablations, and representational analyses of HypRQ-VAE.The analyses particularly examine the model’s effectiveness for long-tail recommendation.

A. Experimental Settings

The study evaluates HypRQ-VAE on three real-world recommendation benchmarks using sequential and generative baselines under standardized ranking protocols. Experiments use constrained decoding and a two-phase training setup.

Datasets: The benchmarks are MovieLens, Instruments, and Arts from Amazon, with item titles and descriptions used as textual content.MovieLens items are additionally augmented with plot overviews from TMDB.
Datasets: Users and items with fewer than five interactions are filtered, histories are ordered chronologically, and sequences are truncated to length 20.H20 and T80 denote the average proportions of head and tail items in user preferences.
Baselines: Baselines include matrix factorization, convolutional and self-attention sequential recommenders, and LLM-based models using textual, collaborative, or learned identifiers.The comparison includes MF, Caser, SASRec, P5-TID, and P5-CID.
Evaluation Protocol: Evaluation uses leave-one-out full-item ranking with Hit@K and NDCG@K for K∈{5,10}, beam size 20, and Trie-based constrained decoding for generative models.The Trie restricts autoregressive generation to valid successor tokens.
Implementation: HypRQ-VAE is trained in two phases, using a four-layer codebook with 256 codes per layer and LoRA fine-tuning for the LLaMA2-7B generative backbone.The supplied implementation details specify the codebook and backbone configuration.
Prompt Design: Prompts combine chronological semantic IDs from interaction histories with natural-language instructions, and responses contain the target item’s semantic ID.The prompt asks the model to predict the next item most likely to engage the user.

B. Overall Performance

Across three datasets, HypRQ-VAE consistently outperforms baseline methods, with gains attributed to hyperbolic space’s greater representational flexibility for long-tail items.

HypRQ-VAE consistently outperforms all baselines across the three evaluated datasets.The overall comparison is reported in Table I.
Figure 5 compares RQ-VAE and HypRQ-VAE top-5 predictions by their head- and tail-item composition across three datasets.
Hyperbolic space’s exponentially expanding capacity provides greater representational flexibility for long-tail items than Euclidean embeddings.

C. Analysis of Long-Tail Performance

HypRQ-VAE improves both head- and tail-item recommendation, with especially large tail-item gains linked to more balanced hyperbolic quantization. Performance depends on semantic-ID length and codebook dimension, with overly large settings reducing accuracy.

Performance on Head vs. Tail Items: +52.71% Hit@10 on MovieLens tail items exceeds the +14.09% gain for head items.HypRQ-VAE improves both groups, but the largest improvement is observed for tail items.
Performance on Head vs. Tail Items: HypRQ-VAE recommends a higher proportion of tail items than the Euclidean model regardless of the ground-truth item’s popularity.This pattern is reported for top-5 recommendation lists and is interpreted as greater item diversity.
Quantifying Representation Quality: Hyperbolic quantization substantially reduces head–tail representation disparity, and MovieLens tail items achieve lower AQE than head items.AQE measures the distance between an item embedding and its assigned codebook vector; absolute AQE values are not directly comparable across geometries.
Length of Semantic IDs: Performance rises as semantic-ID length increases from 3 to 6 but declines from 6 to 8, consistent with limited detail at short lengths and error accumulation at long lengths.
Codebook Dimension: Increasing codebook dimension generally improves performance, but excessively large codebooks can reduce it through noise sensitivity and overfitting.The tested dimensions are 64, 128, 256, and 512.

V. RELATED WORK

Prior generative recommendation methods use diverse item-indexing strategies, but codebook approaches largely remain Euclidean. HypRQ-VAE introduces hyperbolic semantic-ID indexing for generative recommendation and reports gains linked to long-tail representation.

Generative recommendation methods use ID-based, textual, codebook-based, multimodal, and parallel-decoding item identifiers.
Existing codebook methods operate mostly in Euclidean space, which struggles with recommendation data’s hierarchical power-law structure.
Hyperbolic approaches have previously improved preference modeling in traditional collaborative filtering.
HypRQ-VAE is presented as the first framework integrating hyperbolic geometry into residual-quantized autoencoders for semantic-ID item indexing.
Experiments report consistent gains over Euclidean methods, particularly for long-tail recommendation, attributed to more balanced quantization.

APPENDIX

Appendix analyses examine head–tail composition, code-path utilization, initialization, and dimensional capacity. HypRQ-VAE produces more tail-oriented recommendations and more balanced token distributions, while later layers may require additional capacity.

Performance Analysis on Head vs. Tail Items: Table V compares head- and tail-item proportions in top-k recommendations under head or tail ground-truth conditions.GTH/GTT denote head/tail ground-truth items, while RH/RT denote predicted head/tail proportions.
Performance Analysis on Head vs. Tail Items: HypRQ-VAE predicts a higher proportion of tail items than RQ-VAE across datasets, top-5 and top-10 predictions, and both ground-truth groups.
Visualization of Codebook Utilization: HypRQ-VAE achieves a more balanced token distribution across codebook layers than the compared methods without additional regularization.
Initialization and Capacity: Uniform initialization produces distinctive per-layer code distributions, and deeper layers require additional representational space for finer-grained information.