PaperlayerView exact revision on arXiv

Source-linked AI summary

ONCE: Boosting Content-based Recommendation with Both Open- and Closed-source Large Language Models

Qijiong Liu, Nuo Chen, Tetsuya Sakai, Xiao-Ming Wu

arXiv:2305.06566v4cs.IRcs.CL

TL;DR

Content-based recommenders can struggle to represent item content accurately. ONCE combines open-source LLM fine-tuning with closed-source LLM prompting, and evaluations report improvements over existing state-of-the-art recommendation models.

  • Problem

    Early word-embedding encoders can miss crucial terms, while small pretrained language models may fail to capture item content and similarity accurately.

    Source: §1 · p. 2

  • Method

    ONCE uses open-source LLMs as fine-tuned content encoders and closed-source LLM prompting to produce enriched training data.

    Source: §1 · pp. 2–3

  • Results

    Both LLM types were effective across the MIND and Goodreads benchmarks, with LLaMA fine-tuning consistently improving by more than 10 percentage points over existing state-of-the-art models.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Combining open-source LLM fine-tuning and closed-source prompting can substantially improve recommendation performance, and ONCE can apply to content-based domains beyond news and books.

    Source: §7 · p. 9

    In the reported experiments, LLaMA was used as a content encoder without prompting.

    Source: §1 · p. 2

Abstract

from arXiv · show

Personalized content-based recommender systems have become indispensable tools for users to navigate through the vast amount of content available on platforms like daily news websites and book recommendation services. However, existing recommenders face significant challenges in understanding the content of items. Large language models (LLMs), which possess deep semantic comprehension and extensive knowledge from pretraining, have proven to be effective in various natural language processing tasks. In this study, we explore the potential of leveraging both open- and closed-source LLMs to enhance content-based recommendation. With open-source LLMs, we utilize their deep layers as content encoders, enriching the representation of content at the embedding level. For closed-source LLMs, we employ prompting techniques to enrich the training data at the token level. Through comprehensive experiments, we demonstrate the high effectiveness of both types of LLMs and show the synergistic relationship between them. Notably, we observed a significant relative improvement of up to 19.32% compared to existing state-of-the-art recommendation models. These findings highlight the immense potential of both open- and closed-source of LLMs in enhancing content-based recommendation systems. We will make our code and LLM-generated data available for other researchers to reproduce our results.

1 INTRODUCTION

ONCE investigates how open- and closed-source LLMs can improve content-based recommendations through different strategies. The paper reports gains from fine-tuning open-source LLMs and complementary benefits from closed-source LLM-generated training data.

1 INTRODUCTION

  • Content-based recommenders rely on item-text encoders; earlier models used CNNs with pretrained word representations [43], while later methods used Transformer PLMs [46].The paper motivates improved recommendation as digital content expands and users need precise, pertinent recommendations.Source: §1 · p. 1
  • Earlier encoders struggled to capture item content: word embeddings missed terms such as “Al-Rassan,” while small PLMs lacked relevant knowledge and sufficient representation dimensions.The paper gives 768 dimensions as an example of the PLMs’ limited representation size.Source: §1 · p. 2
  • LLaMA-generated representations reflected the books’ content more accurately, placing “The Lions of Al-Rassan” and “The Summer Tree” as more similar to each other than either was to “The Lion King.”Source: §1 · p. 2
  • ONCE uses DIRE to fine-tune open-source LLMs as content encoders and GENRE prompting to enrich training data with closed-source LLM outputs.In experiments, LLaMA was used as a content encoder without prompting.Source: §1 · p. 2
  • Fine-tuning LLaMA consistently improved recommendation results by more than 10 percentage points over existing state-of-the-art recommendation models.Evaluations used the MIND and Goodreads benchmarks.Source: §1 · p. 3
  • ChatGPT-enriched data complemented LLaMA fine-tuning by improving recommendation quality while substantially accelerating training efficiency.Source: §1 · p. 3

2 OVERVIEW

The overview defines content-based recommendation as inferring a user’s interest in candidate content from click data and describes a three-module model. ONCE combines closed-source prompting with open-source LLM content encoding to enhance recommendations.

2.1 Content-based Recommendation

  • Content-based recommendation infers a user’s interest in candidate content from click data that records user, content, and click label.Each user has a history of browsed content, while each content item can have features such as title, category, or description.Source: §2.1 · p. 3
  • A typical model encodes content features and browsing histories into d-dimensional vectors, then classifies which candidate best aligns with the user vector.Its three modules are a content encoder, a history encoder, and an interaction module.Source: §2.1 · p. 3

2.2 Enhancing Content-based Recommendation with Open- and Closed-source LLMs (ONCE)

  • ONCE uses closed-source prompting in GENRE to enrich recommendation data with external knowledge, then uses open-source LLM deep layers in DIRE as content encoders.The framework combines the two LLM types to strengthen the recommendation system.Source: §2.2 · p. 3

3 DIRE: FINETUNING OPEN-SOURCE LLMS

DIRE uses open-source LLMs as content encoders and combines natural-language content formatting, attention-based fusion, and a compute-conscious fine-tuning strategy for personalized recommendation.

3 DIRE: FINETUNING OPEN-SOURCE LLMS

  • DIRE integrates an open-source LLM and attention fusion into content-based recommendation to improve performance through its fine-tuning strategy.The section frames fine-tuning strategy as a complement to the pretrained model’s knowledge and capabilities.Source: §3 · p. 4, §3.1 · p. 4

3.1 Network Architecture

  • Natural-language templates concatenate item fields, and attention fusion condenses LLM hidden states into a content representation for personalized recommendation.The resulting representation is passed to the user-modeling or interaction module.Source: §3.1 · p. 4

3.2 Finetuning Strategy

  • The approach fixes lower LLM layers, precomputes and stores their dataset hidden states, fine-tunes only the uppermost k layers (H ≫ k), and applies LoRA to unfrozen Transformer layers.For LLaMA-7B with 32 layers, only the top 2 are fine-tuned; caching reduces LLM computation load to 2/32 ≈ 6% of original cost.Source: §3.2 · p. 4

4 GENRE: PROMPTING CLOSED-SOURCE LLMS

GENRE uses GPT-3.5 to augment training data for conventional recommendation models, addressing the limits of direct LLM recommendation through content summarization, user profiling, and personalized content generation. Its chain-based variant can iteratively update generated data and use inferred user interests to better tailor synthetic content.

4 GENRE: PROMPTING CLOSED-SOURCE LLMS

  • Prior direct use of closed-source LLMs as recommenders only matched basic matrix factorization or random recommendations, falling short of modern attention-based approaches.Earlier approaches used prompting or in-context learning without fine-tuning.Source: §4 · p. 4
  • GENRE prompts GPT-3.5 to generate data, optionally updates it for another generation round, and trains conventional news recommendation models on the generated data.The four steps are prompting, generating, optional updating, and training; generation without updating is one-pass, while updating is chain-based.Source: §4 · p. 4, Figure 4 · p. 5

4.1 LLMs as Content Summarizer

  • Given a content title, abstract, and category, GPT-3.5 generates a more informative title that replaces the original as a content-encoder input.LLMs can associate infrequent people and locations with pretraining knowledge, helping address sparse representations.Source: §4.1 · p. 5

4.2 LLMs as User Profiler

  • GPT-3.5 infers topics and regions from browsing history, combines their pooled embeddings with the history-encoder user vector, and uses the result instead for click prediction.User profiles are generally absent from anonymized datasets for privacy; GPT-3.5 can infer interests such as Florida from a mention of Miami.Source: §4.2 · p. 5

4.3 LLMs as Personalized Content Generator

  • GPT-3.5 generates synthetic content tailored to users with limited browsing histories, augmenting interactions so the history encoder can learn more effective user representations.Source: §4.3 · p. 5

4.4 Chain-based Generation

  • The chain-based generator first infers topics and regions, then prompts GPT-3.5 with that profile and user history to generate content better matched than one-pass prompting.Generated data can update current data and be reused in a subsequent prompting and generation round.Source: §4.4 · p. 5

5 EXPERIMENTS

Across MIND and Goodreads, ONCE combines open- and closed-source LLMs for content-based recommendation, with the experiments examining benchmark performance and the effects of model tuning and synthetic content. The reported findings show gains from both LLM types and from adding generated content for new and warm users.

5.1 Experimental Setup

  • Experiments use the MIND news and Goodreads book datasets and evaluate ONCE with NAML, NRMS, and Fastformer, comparing against PLM-NR.The study uses LLaMA-7B and LLaMA-13B as open-source models and GPT-3.5 as its closed-source model.Source: §5.1 · pp. 5–6

5.2 Performance Comparison

  • 25%: ONCE matches LLaMA-13B’s NRMS performance at epoch 8 by epoch 6; on Fastformer, it exceeds the epoch-15 performance by epoch 9, a 40% enhancement.Figure 5 shows training curves for open-source LLMs and ONCE; ONCE uses LLaMA-13B as its backbone.Source: Figure 5 · p. 6, §5.2 · p. 6

5.3 Ablation Study on Open-source LLMs

  • Across MIND and Goodreads, untuned LLMs often improve recommenders, but BERT on Goodreads is an exception because book titles offer limited enriched knowledge.LLaMA-7B generally outperforms LLaMA-13B on MIND when tuning 1–2 layers, while LLaMA-13B has the most promising Goodreads results; tuning more layers generally improves performance but raises training costs.Source: §5.3 · pp. 6–7

5.2 Performance Comparison

  • Open-source, closed-source, and dual-LLM enhancements improve base recommender performance, with ONCE gaining further over a single LLM, though less than open-source fine-tuning.The authors attribute open-source gains to semantic knowledge in continuous embeddings and closed-source gains to enriched token-level features; combining them adds supplementary semantic insight.Source: §5.2 · p. 6, Table 2 · p. 7

5.3 Ablation Study on Open-source LLMs

  • LoRA improves open-source LLM performance on MIND, but Goodreads shows a different pattern, where title-only inputs require more nuanced parameter adjustments.The proposed explanation contrasts news headlines, which capture content essence, with less informative book titles.Source: §5.3 · p. 7

5.4 Ablation Study on Closed-source LLMs

  • Personalized generated content improves performance for both new and warm users in most cases by helping the history encoder capture interests despite limited user history.The authors explain that adding generated content to new users’ histories can also improve the encoder’s ability to model warm users.Source: §5.4 · p. 7

6 RELATED WORKS

The related work situates LLM use in recommendation by pipeline role and discusses content-based recommendation, including the paper’s focus on news and books. It presents the combination of open- and closed-source LLMs as a contribution amid limited study of LLM-based news recommendation.

6.1 LLMs for Recommendation

  • LLM recommendation is classified by pipeline position or tuning and CRM integration; this work enhances datasets and encodes content, reporting the first combination of open- and closed-source LLMs in recommendation [22].The pipeline categories are user data collection, feature engineering, feature encoder, scoring or ranking, and pipeline controller.Source: §6.1 · p. 9

6.2 Content-based Recommendation

  • Content-based recommendation spans music, news [12], and video, while this work focuses on news and books.News research includes deep neural models [43] [44] and PLM-based systems [46] [48]; although LLMs have been applied to personalization [31] and product recommendation [19], direct LLM recommendation has shown negative results [25].Source: §6.2 · p. 9

7 CONCLUSION · A PROMPTS FOR CLOSED-SOURCE LLMS

ONCE combines open-source LLM fine-tuning and closed-source LLM prompting to improve content-based recommendation, and the appendix documents prompts for its closed-source components. The framework is intended to extend beyond news and book recommendation.

7 CONCLUSION

  • Combining open-source LLM fine-tuning with closed-source LLM prompting can substantially improve content-based recommendation systems.The paper presents ONCE as applicable to content-based domains beyond news and book recommendation.Source: §7 · p. 9
  • ONCE is presented as applicable to content-based domains beyond news and book recommendation.The authors hope the work will encourage further research and contribute to more effective LLM-based recommendation systems.Source: §7 · p. 9

A PROMPTS FOR CLOSED-SOURCE LLMS

B MORE EXPERIMENTS FOR CLOSED-SOURCE LLMS · B.1 More base models

The appendix adds prompt-based closed-source LLM experiments using OpenAI GPT-3.5. It evaluates GPT-generated data with matching-based recommendation models and ranking-based deep CTR models.

B MORE EXPERIMENTS FOR CLOSED-SOURCE LLMS

B.1 More base models

  • GPT-generated data is evaluated with four matching-based recommenders: NAML [43], LSTUR, NRMS, and PLMNR [46].Source: §B.1 · p. 11
  • The evaluation also includes four ranking-based deep CTR models: BST [5], DCN [40], PNN [28], and DIN [53].Source: §B.1 · p. 11

B.2 Performance Comparison

On MIND, combining the content summarizer with the chain-based personalized content generator performs best in most cases and significantly outperforms training on original data. The chain-based generator also performs better than one-pass variants.

B.2 Performance Comparison

  • The ALL combination achieves the best performance for all recommendation models in most cases and significantly outperforms training with original data (ORI).ALL combines content titles from the one-pass content summarizer with synthetic content from the chain-based personalized content generator.Source: §B.2 · p. 12, Table 6 · p. 12
  • The chain-based personalized content generator performs better than one-pass variants, indicating the effectiveness of the chain-of-thought prompt.Source: §B.2 · p. 12, Table 6 · p. 12
  • The section also compares generative schemes by cost and cost conversion rate (CCR), defined as improvement in AUC relative to cost.Source: Table 7 · p. 12

B.3 Content Summarizer · B.4 Personalized Content Generator

Adding abstracts and subcategories improves recommendation performance, while GPT-3.5-generated news titles outperform using all available news features. Across four base models, recommendation performance increases as more personalized news articles are generated.

B.3 Content Summarizer

  • The Goodreads dataset uses only book titles as text features, whereas MIND provides news titles, abstracts, and categories.Source: §B.3 · p. 12
  • The preceding experiments used enhanced news titles and categories as content-encoder inputs; this subsection assesses the impact of combining more news features.Source: §B.3 · p. 12
  • Adding abstracts and subcategories improves model performance, although existing models often exclude these features for efficiency.The additional features are news abstracts and subcategories.Source: §B.3 · p. 13
  • MIND-NS* outperforms MIND* despite MIND* including all available news features, supporting the effectiveness of GPT-3.5-generated news titles.Source: §B.3 · p. 13

B.4 Personalized Content Generator

  • For four base models on MIND, the evaluation compared using 0, 1, or 2 generated news articles per new user.Source: §B.4 · p. 13
  • For each of the four base models, recommendation performance improves as the number of generated news articles increases.Source: §B.4 · p. 13

B.5 Cost Conversion Rate

Under ONCE, the best cost conversion rate depends on the model type: CG leads for matching models, while CS leads for ranking models. UP has the worst CCR, and chain-based generation improves more than one-pass generation but at lower CCR.

B.5 Cost Conversion Rate

  • Chain-based generation achieves higher improvement than one-pass generation, but its CCR decreases because it uses the expensive UP.Source: §B.5 · p. 14
  • With the full dataset, CG has the best CCR for matching-based models, while CS has the best CCR for ranking-based models.Source: §B.5 · p. 14
  • UP has the worst CCR because long browsing histories increase profiler request token counts and costs.Source: §B.5 · p. 14

Related papers

Loading 2305.06566v4…