PaperlayerView exact revision on arXiv

Source-linked AI summary

HLLM: Enhancing Sequential Recommendations via Hierarchical Large Language Models for Item and User Modeling

Junyi Chen, Lu Chi, Bingyue Peng, Zehuan Yuan

arXiv:2409.12740v1cs.IRcs.AI

TL;DR

LLM recommenders have so far offered modest gains, and the value of pre-training, task-specific fine-tuning, and scalability in recommendation remains underexplored. HLLM uses one LLM to extract item features and another to model users’ interaction histories; it reports state-of-the-art academic results and validates practical applicability online.

  • Problem

    Existing LLM-based methods have not significantly improved on traditional recommenders, and the value of pre-trained weights, recommendation fine-tuning, and scalability remain underexplored.

    Source: §1 · p. 2

  • Method

    HLLM uses an Item LLM to extract item features and a User LLM to model user interests and predict future behavior from those features.

    Source: Hierarchical Large Language Model… · p. 3

  • Results

    HLLM achieves state-of-the-art results on academic datasets, with reported gains from pre-trained weights, recommendation fine-tuning, and increasing model size up to 7 billion parameters.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Real-world online A/B testing validates HLLM’s practical efficiency and applicability.

    Source: §5 · p. 8

Abstract

from arXiv · show

Large Language Models (LLMs) have achieved remarkable success in various fields, prompting several studies to explore their potential in recommendation systems. However, these attempts have so far resulted in only modest improvements over traditional recommendation models. Moreover, three critical questions remain under-explored: firstly, the real value of LLMs' pre-trained weights, often considered to encapsulate world knowledge; secondly, the necessity of fine-tuning for recommendation tasks; lastly, whether LLMs can exhibit the same scalability benefits in recommendation systems as they do in other domains. In this paper, we propose a novel Hierarchical Large Language Model (HLLM) architecture designed to enhance sequential recommendation systems. Our approach employs a two-tier model: the first Item LLM extracts rich content features from the detailed text description of the item, while the second User LLM utilizes these features to predict users' future interests based on their interaction history. Extensive experiments demonstrate that our method effectively leverages the pre-trained capabilities of open-source LLMs, and further fine-tuning leads to significant performance boosts. Additionally, HLLM achieves excellent scalability, with the largest configuration utilizing 7B parameters for both item feature extraction and user interest modeling. Moreover, HLLM offers excellent training and serving efficiency, making it practical in real-world applications. Evaluations on two large-scale datasets, PixelRec and Amazon Reviews, show that HLLM achieves state-of-the-art results, outperforming traditional ID-based models by a wide margin. In online A/B testing, HLLM showcases notable gains, validating its practical impact in real-world recommendation scenarios. Codes are available at https://github.com/bytedance/HLLM.

1 Introduction

Prior LLM-based recommender gains have been modest, while the value of pre-training, recommendation fine-tuning, and LLM scalability remain open questions. HLLM separates item feature extraction from user-interest modeling and reports strong academic and online results.

1 Introduction

  • Prior sequential recommenders rely heavily on IDs and shallow networks, limiting cold-start performance and the modeling of complex, diverse interests.LLM-based methods also show non-significant improvements over traditional methods and can incur long sequences and costly generation.Source: §1 · pp. 1–2
  • The paper identifies three unresolved questions: the value of pre-trained weights, the need for recommendation fine-tuning, and whether LLM scalability extends to recommendation.It notes that scalability for models exceeding 1 billion parameters remained an open question in this domain.Source: §1 · p. 2
  • HLLM extracts item features from text with an Item LLM, then uses a User LLM to model interests and predict future behavior from those features.The resulting embeddings shorten behavior sequences and reduce computational complexity relative to text-based LLM recommenders.Source: §1 · p. 2
  • Experiments find pre-trained weights beneficial for item extraction and user-interest modeling, while recommendation-specific fine-tuning is crucial for surpassing traditional methods.The authors describe the pre-trained weights as transferring LLM world knowledge into recommendation.Source: §1 · p. 2
  • HLLM performance improves with more data and larger models, with consistent gains reported up to 7 billion parameters within available resources.The authors also report a training-efficiency advantage over ID-based models, surpassing them with a small amount of training data.Source: §1 · p. 2
  • HLLM achieves state-of-the-art results across academic datasets and is validated through real-world online A/B testing.The authors also report training and serving efficiency.Source: §1 · p. 2

2 Related Work

Traditional recommenders largely use ID embeddings and develop feature interactions or model historical interests. Language-model approaches range from information augmentation and dialogue formats to adapting LLMs for recommendation inputs and outputs.

Traditional Recommender Systems

  • Traditional recommenders predominantly use ID-based embeddings, with feature interactions as a central design concern.DeepFM combines FM for low-order and DNN for high-order interactions; DCN explicitly crosses features across layers.Source: Traditional Recommender Systems · p. 2
  • DIN and DIEN use attention to capture diverse user interests, while SASRec applies self-attention to sequential recommendation.CLUE and HSTU demonstrate scaling behavior with parameter counts in the hundreds of millions.Source: Traditional Recommender Systems · p. 2

Recommendation with Language Models

  • LLM recommendation work includes summarizing or supplementing user and item information, conversational formats, and adapting LLMs beyond text-only inputs and outputs.Examples include LLM-generated cold-start training signals and hybrid ID-embedding plus textual-feature prompts.Source: Recommendation with Language Models · pp. 2–3
  • HLLM uses separate Item and User LLMs: the first turns item descriptions into embeddings, and the second uses historical item embeddings to predict the next item.The figure describes the two LLMs as having non-shared parameters and being optimized through next-item prediction.Source: Figure 1 · p. 3
  • Existing LLM recommendation methods are reported to offer limited improvements over traditional recommendation models.Source: Recommendation with Language Models · p. 3

3 Method

Section 3 formulates sequential recommendation as next-item prediction and introduces HLLM, which separates item feature extraction from user-interest modeling. It aligns the architecture with generative and discriminative recommendation objectives.

Problem Formulation

  • Given a user's chronologically ordered interaction history, the task is to predict the next item using item text information.The method uses item text rather than item IDs.Source: Problem Formulation · p. 3

Hierarchical Large Language Model Architecture

  • HLLM decouples item modeling from user modeling, compressing item descriptions into embeddings before modeling the user profile with a second LLM.The architecture is designed to reduce the burden of long text histories while making minimal structural changes to pretrained LLMs.Source: Hierarchical Large Language Model… · p. 3

Item LLM

  • The Item LLM appends a special [ITEM] token to an item's textual attributes and uses that token's final-layer hidden state as the item embedding.The text is flattened into a sentence, preceded by a fixed prompt, and tokenized before [ITEM] is appended.Source: Item LLM · p. 3

User LLM

  • The User LLM consumes the sequence of Item LLM embeddings and predicts the next item embedding; its inputs and outputs are embeddings rather than text.The method discards the pretrained word embeddings while retaining the other pretrained weights.Source: User LLM · pp. 3–4

Training for Recommendation Objectives

  • HLLM uses supervised fine-tuning on pretrained LLMs and supports generative or discriminative recommendation through adjusted training objectives.The paper notes that the two recommendation categories require appropriate objective adjustments.Source: Training for Recommendation Objectives · p. 4

Generative Recommendation

  • Generative recommendation predicts the next item embedding from prior embeddings and trains with InfoNCE, using the target as positive and randomly sampled items as negatives.The loss uses a similarity function with a learnable temperature parameter.Source: Generative Recommendation · p. 4

Discriminative Recommendation

  • For discriminative recommendation, early fusion inserts the target embedding into the history, while late fusion combines target and user embeddings in a prediction head.Early fusion tends to perform better but is harder to apply across many candidates; late fusion shares user features across candidates and is more efficient, typically with lower performance.Source: Discriminative Recommendation · p. 4

4 Experiments

Experiments on PixelRec and Amazon Book Reviews evaluate HLLM’s pre-training and fine-tuning, scalability, benchmark performance, efficiency, and online impact. HLLM outperforms the compared models, improves with model and data scale, and produces a gain in online ranking metrics.

Datasets and Evaluation Setup

  • Offline evaluation uses leave-one-out splits and Recall@K and NDCG@K on PixelRec subsets and Amazon Book Reviews.The open-source datasets are used only for offline training and evaluation.Source: Datasets and Evaluation Setup · p. 5

Baselines and Training

  • HLLM uses TinyLlama-1.1B or Baichuan2-7B for both LLMs, while comparison models include SASRec-1B and HSTU-1B.Resource constraints limited HLLM training to five epochs; other models were trained for 50 and 200 epochs, respectively.Source: Baselines and Training · p. 5

Pre-training and Fine-tuning (RQ1)

  • Pre-trained weights benefit both item feature extraction and user interest modeling, while fine-tuning both LLMs is crucial to outperforming ID-based models.Freezing either the Item LLM or the pre-trained User LLM yields critically low performance; conversation-data SFT can have slight negative effects.Source: Pre-training and Fine-tuning (RQ1) · pp. 5–6

Scaling Up (RQ2)

  • Increasing the parameters of either LLM improves performance, and scaling both from 1B to 7B further improves results on Amazon Books.Across PixelRec data volumes from 0.1M to 8M, performance improves with more data and no bottleneck is observed at the tested scale.Source: Scaling Up (RQ2) · p. 6, Figure 3 · p. 6

HLLM vs. SOTA Methods (RQ3)

  • Scaling HLLM from 1B to 7B continues to improve recommendation performance, while larger ID-based models show modest gains or declines.On Pixel8M, HSTU-large gains 0.76 in R@200 versus HLLM-1B’s 2.44 under the same setting.Source: HLLM vs. SOTA Methods (RQ3) · p. 6

Training and Serving Effeciency (RQ4)

  • HLLM requires only one-sixth to one-fourth of the data volume to match ID-based methods’ performance.Item caching reduces the cost of encoding items in advance; despite some metric decreases, cached HLLM still exceeds ID-based models.Source: Training and Serving Effeciency (RQ4) · pp. 6–7

HLLM vs. SOTA Methods (RQ3)

  • 22.93% average improvement on Pixel8M and 108.68% on Books for HLLM-1B versus the lowest-performing baseline; HLLM-7B improves 169.58% versus the baseline.HLLM decisively outperforms other models across metrics and datasets.Source: HLLM vs. SOTA Methods (RQ3) · p. 6, Table 7 · p. 7

Online A/B Test

  • Online ranking experiments report a 0.705% increase in key metrics.The online system extracts item embeddings at item creation, updates active users’ embeddings daily, and has virtually unchanged inference time.Source: Online A/B Test · p. 7, Figure 4 · p. 7

5 Conclusion

The conclusion presents HLLM as a hierarchical architecture for sequential recommendation that uses LLMs for item features and user-interest modeling. It reports strong scalability, state-of-the-art academic-dataset results, and online A/B-test validation.

5 Conclusion

  • HLLM uses LLMs to extract item features and model user interests in sequential recommendation.The conclusion states that recommendation-objective fine-tuning is essential.Source: §5 · p. 8
  • HLLM exhibits scalability with larger model parameters and outperforms traditional ID-based models on academic datasets.The paper characterizes its results as state of the art.Source: §5 · p. 8
  • Online A/B testing further validates HLLM’s practical efficiency and applicability.Source: §5 · p. 8

A More Experiments on Academic Datasets

The item-text ablation finds that richer text and longer input improve performance by supplying the Item LLM with more detailed information to distinguish items and support user-interest modeling.

A More Experiments on Academic Datasets

Method of Item LLM Feature Extraction

For item feature extraction, the paper appends a special [ITEM] token to the text input and uses its output as the item representation. This approach outperforms mean pooling of the LLM’s final-layer hidden states.

Method of Item LLM Feature Extraction

Sequence Length of User LLM

Increasing User LLM sequence length benefits HLLM, though gains are modest on the academic dataset and stable improvements appear in industrial systems with longer histories.

Sequence Length of User LLM

Compatibility with ID-based Features

HLLM can combine item-description features with ID-based inputs. In the validation, timestamps improved performance markedly, while raw item IDs slightly reduced it.

Compatibility with ID-based Features

B Scaling Up of HLLM on Industrial Dataset

Industrial experiments examine HLLM scalability, including the effect of longer user histories. Performance steadily increases as the User LLM sequence length grows.

B Scaling Up of HLLM on Industrial Dataset

Sequence Length of User LLM

  • HLLM performance steadily increases as industrial user-sequence length grows, indicating potential for modeling longer histories.Table 13 reports the sequence-length experiments.Source: Sequence Length of User LLM · p. 11

Parameters of Item LLM and User LLM

On the industrial dataset, AUC consistently increases as the parameter counts of both the Item LLM and User LLM grow.

Parameters of Item LLM and User LLM

Loading 2409.12740v1…