PaperlayerView exact revision on arXiv

Source-linked AI summary

ONCE: Boosting Content-based Recommendation with Both Open- and Closed-source Large Language Models

Qijiong Liu, Nuo Chen, Tetsuya Sakai, Xiao-Ming Wu

arXiv:2305.06566v4cs.IRcs.CL

TL;DR

Content-based recommenders struggle to understand item content as digital collections expand. ONCE combines open-source LLM finetuning for content representations with closed-source LLM prompting for richer training data, and reports substantial gains across recommendation benchmarks. The findings support using both LLM types together while noting that the open-source encoder experiments used LLaMA without prompting.

  • Problem

    Existing content-based recommenders struggle to fully understand item content, limiting semantic representations for personalized recommendation.

  • Method

    ONCE finetunes open-source LLMs as content encoders and prompts closed-source LLMs to enrich textual and user training features.

  • Results

    Across MIND and Goodreads, both LLM types were effective, with LLaMA finetuning improving performance by more than 10 percentage points over existing state-of-the-art recommendation models.

  • Takeaways & Limitations

    Combining open-source LLM finetuning with closed-source LLM prompting can substantially improve content-based recommendation and support broader content-based domains.

  • Takeaways & Limitations

    The experiments used LLaMA as a content encoder without prompting.

Abstract

from arXiv · show

Personalized content-based recommender systems have become indispensable tools for users to navigate through the vast amount of content available on platforms like daily news websites and book recommendation services. However, existing recommenders face significant challenges in understanding the content of items. Large language models (LLMs), which possess deep semantic comprehension and extensive knowledge from pretraining, have proven to be effective in various natural language processing tasks. In this study, we explore the potential of leveraging both open- and closed-source LLMs to enhance content-based recommendation. With open-source LLMs, we utilize their deep layers as content encoders, enriching the representation of content at the embedding level. For closed-source LLMs, we employ prompting techniques to enrich the training data at the token level. Through comprehensive experiments, we demonstrate the high effectiveness of both types of LLMs and show the synergistic relationship between them. Notably, we observed a significant relative improvement of up to 19.32% compared to existing state-of-the-art recommendation models. These findings highlight the immense potential of both open- and closed-source of LLMs in enhancing content-based recommendation systems. We will make our code and LLM-generated data available for other researchers to reproduce our results.

1 INTRODUCTION

Content-based recommenders need stronger content understanding as existing encoders miss important semantic relationships. ONCE addresses this with open-source LLM representations and closed-source LLM-generated training data, improving recommendation performance and efficiency.

  • 1 INTRODUCTION: The growing volume of digital content increases the need for precise and pertinent personalized recommendations.The motivation covers platforms such as news websites and book recommendation services.
  • 1 INTRODUCTION: Existing content encoders struggle to fully comprehend items, with word embeddings and small PLMs missing crucial terms, knowledge, and content-level similarities.The examples focus on book titles whose semantic relationships are poorly represented by earlier encoders.
  • 1 INTRODUCTION: LLaMA-generated descriptions produced representations that correctly rated the two Guy Gavriel Kay novels as more similar to each other than to The Lion King.The representations were created from concise descriptions generated using title information.
  • 1 INTRODUCTION: ONCE uses DIRE to fine-tune open-source LLMs as content encoders and GENRE prompts to enrich training data with closed-source LLMs.The two strategies target embedding-level content representations and token-level textual and user features, respectively.
  • 1 INTRODUCTION: Finetuning LLaMA improved recommendation performance by more than 10 percentage points over existing state-of-the-art models across evaluations on MIND and Goodreads.The study also reports complementary benefits from ChatGPT-generated data during LLaMA finetuning.

2 OVERVIEW

Content-based recommendation predicts user interest in candidate content from item features, user browsing history, and click data. Its architecture combines content encoding, history encoding, and interaction-based candidate selection, while ONCE incorporates open-source LLMs into this framework.

  • 2 OVERVIEW: Content-based recommendation infers a user’s interest in candidate content from item features, browsing history, and click labels.Contents may include titles, categories, or descriptions, and each click records whether the user selected a candidate.
  • 2 OVERVIEW: A typical model uses a content encoder, history encoder, and interaction module to represent items, summarize browsing sequences, and select the best-aligned candidate.The content and user representations are unified as d-dimensional vectors before interaction.
  • 2 OVERVIEW: ONCE leverages open- and closed-source LLMs to enhance content-based recommendation through their pretrained semantic understanding and knowledge.The overview presents the framework as addressing a gap between open-source and closed-source model capabilities.

3 DIRE: FINETUNING OPEN-SOURCE LLMS

DIRE integrates an open-source LLM into content-based recommendation by encoding natural-language item sequences, fusing hidden states into content representations, and fine-tuning selected layers for recommendation.

  • DIRE incorporates an open-source LLM and attention fusion layer into the content-based recommendation framework.The resulting content representation is passed to user modeling or interaction modules for personalized recommendation.
  • 3.1 Network Architecture: Natural language templates concatenate multiple content fields into a cohesive sequence before LLM embedding.The approach labels the overall sequence and prefixes individual features, rather than using special field-separation tokens.
  • 3.1 Network Architecture: Transformer layers pass hidden states sequentially, producing representations across the LLM depth.The model has H Transformer layers, with each layer receiving the preceding layer’s output.
  • 3.1 Network Architecture: An attention fusion layer maps high-dimensional hidden states from d_n to d dimensions and condenses them into a unified representation.A learnable linear transformation performs the dimensionality reduction before additive attention produces the final representation.
  • 3.2 Partial Freezing and Caching: The framework keeps lower LLM layers fixed and fine-tunes only the uppermost k layers, with H≫k, while caching hidden states.This design addresses the computational demands of running large language models.
  • 3.3 Parameter-Efficient Tuning: DIRE applies LoRA to unfrozen Transformer layers to reduce the number of trainable parameters.The paper also tests fine-tuning without LoRA.

4 GENRE: PROMPTING CLOSED-SOURCE LLMS

GENRE uses prompting with closed-source LLMs to generate and optionally update data for conventional recommendation models. Its applications include enriching content, profiling users, generating personalized content, and chaining these generation steps.

  • 4 GENRE: PROMPTING CLOSED-SOURCE LLMS: GENRE prompts closed-source GPT-3.5 to generate data that augments downstream conventional recommendation models.The workflow comprises prompting, generating, and optional updating; updating distinguishes Chain-based from One-pass Generation.
  • 4 GENRE: PROMPTING CLOSED-SOURCE LLMS: GENRE’s framework is presented in Figure 4a, while Figure 4b illustrates prompting GPT-3.5 as a data augmenter.The figure distinguishes the overall framework from a concrete prompting operation.
  • 4.1 LLMs as Content Summarizer: LLMs summarize content into more informative titles that replace original titles as content-encoder inputs.The prompts use a content title, abstract, and category to produce the enhanced title.
  • 4.2 LLMs as User Profiler: LLM-generated user topics and regions are fused into an interest vector and combined with the history-based user vector.The interest-aware user vector then replaces the original user vector for click-probability prediction.
  • 4.3 LLMs as Personalized Content Generator: GPT-3.5 models user-interested content from limited browsing histories by generating synthetic content for new users.The goal is to help the user encoder capture characteristics when interaction data are scarce.
  • 4.4 Chain-based Generation: Chain-based generation first infers user topics and regions, then combines them with user history to prompt better-matched synthetic content.The user profile supports the subsequent generation step.

5 EXPERIMENTS

Experiments on MIND and Goodreads evaluate open-source, closed-source, and combined LLM enhancements across content-based recommendation models, including performance, efficiency, and ablations. The results show strong gains from open-source LLMs, additional benefits from closed-source augmentation, and trade-offs involving tuning depth and training cost.

  • 5.2 Performance Comparison: Open-source LLMs substantially improve four base recommenders across the MIND and Goodreads benchmarks, while closed-source augmentation also improves performance.The comparison includes original recommenders, DIRE, GENRE, and ONCE variants.
  • 5.2 Performance Comparison: Combining open- and closed-source LLMs yields additional gains, although closed-source token-level features contribute less than open-source continuous representations.The reported explanation contrasts token-level data enrichment with embedding-level representations.
  • 5.2 Performance Comparison: ONCE reaches LLaMA-13B’s NRMS eighth-epoch performance by epoch six and surpasses its Fastformer fifteenth-epoch performance by epoch nine.These correspond to reported training-efficiency improvements of 25% and 40%, respectively.
  • Ablation Study on Open-source LLMs: Greater numbers of tuned layers generally improve performance but increase training costs, while optimal model size and tuning depth differ between MIND and Goodreads.LLaMA-7B generally performs better than LLaMA-13B with one to two tuned layers on MIND, whereas LLaMA-13B is strongest on Goodreads.
  • Ablation Study on Open-source LLMs: LoRA improves MIND performance but behaves differently on Goodreads, where informative representations from book titles require more nuanced parameter adjustments.The datasets differ because MIND headlines encode more content than Goodreads book titles.
  • Ablation Study on Closed-source LLMs: Synthetic personalized content improves both new-user and warm-user groups in most cases by helping the history encoder capture interests.The generated content is added to new users’ histories, addressing limited-history modeling difficulty.

6 RELATED WORKS

Related work situates LLM-enhanced recommendation within broader LLM applications and categorizes their roles in recommendation pipelines. Content-based recommendation research spans multiple domains, while news recommendation has progressed from deep neural encoders toward pretrained language models.

  • LLMs in Recommendation: LLM applications in recommendation can be categorized by pipeline position, including user data collection, feature engineering, feature encoding, scoring or ranking, and pipeline control.An alternative categorization considers whether the LLM is tuned and whether it performs inference.
  • Content-based Recommendation: Content-based recommendation research covers music, news, and video domains, while this paper focuses on news and book recommendation.The cited work frames content-based recommendation as a broad application area.
  • News Recommendation: News recommendation models progressed from deep neural networks with limited semantic comprehension toward pretrained language models such as BERT and GPT.The shift is motivated by pretrained language models’ stronger text representation capabilities.

7 CONCLUSION

The work proposes ONCE, which combines open- and closed-source LLMs to improve content-based recommendation. It presents prompts and examples for several closed-source LLM generation schemes used in the framework.

  • ONCE combines open- and closed-source LLMs to enhance content-based recommendation performance.
  • The appendix illustrates prompts for content summarization, user profiling, personalized content generation, and chain-based personalized content generation.The prompts are demonstrated for the MIND dataset.
  • The appendix includes a prompt and example for a content summarizer on the MIND dataset.
  • The appendix describes news-feature inputs used for MIND and MIND-NS, including enhanced titles, images, categories, abstracts, and subcategories.An asterisk denotes the use of additional abstract and subcategory information.
  • The appendix includes a prompt and example for a user profiler on the MIND dataset.

B MORE EXPERIMENTS FOR CLOSED-SOURCE LLMS

Additional experiments evaluate prompt-based closed-source LLMs using GPT-3.5. They test GPT-generated data with established matching-based and ranking-based recommendation models.

  • The experiments assess prompt-based closed-source LLMs, specifically OpenAI GPT-3.5.
  • GPT-generated data is evaluated with four matching-based models: NAML, LSTUR, NRMS, and PLMNR.
  • GPT-generated data is also evaluated with four ranking-based deep CTR models: BST, DCN, PNN, and DIN.

B.2 Performance Comparison

The performance comparison evaluates several GPT-generated content schemes and their combination on MIND. The combined scheme generally performs best, while chain-based generation outperforms one-pass variants.

  • The comparison covers one-pass content summarization, user profiling, personalized generation, chain-based generation, and their combined “ALL” configuration.
  • The study compares AUC, MRR, N@5, and N@10 across the evaluated recommendation models.
  • “ALL” achieves the best performance for most recommendation models and significantly outperforms training with the original data.
  • Chain-based personalized content generation performs better than one-pass variants, indicating the effectiveness of the chain-of-thought prompt.
  • Table 7 compares the cost and cost conversion rate of different generative schemes, including improvement in AUC relative to the original dataset.CCR is defined as the ratio of improvement to cost.

B.3 Content Summarizer

Additional experiments examine how news features and the amount of generated content affect recommendation performance. More features and more generated articles are associated with improved results in the reported comparisons.

  • The content-feature comparison uses news titles, images, and categories, with additional abstract and subcategory inputs in the expanded settings.
  • The personalized content generator prompt asks for a news item based on a user’s browsing history.
  • Adding abstract and subcategory information improves model performance, although existing models often exclude these features for efficiency.
  • MIND-NS* outperforms MIND*, indicating the effectiveness of GPT-3.5-generated news titles.
  • Recommendation performance improves for each of four base models as generated news articles increase from 0 to 1 and 2 per new user.
  • The chain-based personalized content generator is included among the evaluated generation schemes.

B.5 Cost Conversion Rate

The analysis compares the cost efficiency of generative schemes in ONCE using cost conversion rate (CCR), balancing AUC improvement against deployment cost.

  • Chain-based generation improves performance more than one-pass generation, but its CCR is lower because it uses the expensive user profiler.
  • The personalized content generator has the best CCR for matching-based models, while the content summarizer performs best for ranking-based models.
  • The user profiler has the worst CCR because long browsing histories create high token counts and increase profiler costs.
Loading 2305.06566v4…