PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMRec: Large Language Models with Graph Augmentation for Recommendation

Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, Chao Huang

arXiv:2311.00423v6cs.IR

TL;DR

Recommendation systems face sparse implicit feedback and side information that may be noisy, incomplete, or low quality. LLMRec uses LLM-based graph augmentation for interactions, item attributes, and user profiles, with denoising mechanisms for augmented data. On benchmark datasets, it demonstrates superiority over state-of-the-art recommendation techniques.

  • Problem

    Sparse implicit feedback and noisy, incomplete, or low-quality side information limit accurate modeling of user preferences in recommendation systems.

  • Method

    LLMRec augments recommendation graphs through LLM-generated interaction edges, item attributes, and user profiles, then applies noisy-feedback pruning and MAE-based feature enhancement.

  • Results

    LLMRec demonstrates superiority over state-of-the-art recommendation techniques on benchmark datasets.

  • Takeaways & Limitations

    LLM-based graph augmentation can address sparse implicit feedback and low-quality side information within the evaluated recommendation settings.

Abstract

from arXiv · show

The problem of data sparsity has long been a challenge in recommendation systems, and previous studies have attempted to address this issue by incorporating side information. However, this approach often introduces side effects such as noise, availability issues, and low data quality, which in turn hinder the accurate modeling of user preferences and adversely impact recommendation performance. In light of the recent advancements in large language models (LLMs), which possess extensive knowledge bases and strong reasoning capabilities, we propose a novel framework called LLMRec that enhances recommender systems by employing three simple yet effective LLM-based graph augmentation strategies. Our approach leverages the rich content available within online platforms (e.g., Netflix, MovieLens) to augment the interaction graph in three ways: (i) reinforcing user-item interaction egde, (ii) enhancing the understanding of item node attributes, and (iii) conducting user node profiling, intuitively from the natural language perspective. By employing these strategies, we address the challenges posed by sparse implicit feedback and low-quality side information in recommenders. Besides, to ensure the quality of the augmentation, we develop a denoised data robustification mechanism that includes techniques of noisy implicit feedback pruning and MAE-based feature enhancement that help refine the augmented data and improve its reliability. Furthermore, we provide theoretical analysis to support the effectiveness of LLMRec and clarify the benefits of our method in facilitating model optimization. Experimental results on benchmark datasets demonstrate the superiority of our LLM-based augmentation approach over state-of-the-art techniques. To ensure reproducibility, we have made our code and augmented data publicly available at: https://github.com/HKUDS/LLMRec.git

1 INTRODUCTION

LLMRec addresses sparse implicit feedback and unreliable side information by using LLMs to augment recommendation graphs from a natural-language perspective. It augments interactions, item attributes, and user profiles, then applies denoising techniques to improve augmented-data reliability and recommendation accuracy.

  • Motivation: Sparse feedback and cold-start conditions hinder collaborative preference modeling, while heterogeneous side information can introduce distortions.The cited discussion covers item attributes, user-generated content, multimodal features, and risks from irrelevant or complex auxiliary information.
  • Approach: LLMRec uses LLMs to augment user-item interaction edges, item node attributes, and user node profiles using natural-language representations.The framework shifts from ID-based recommendation toward modality-based representations intended to capture richer user-preference context.
  • Robustification: LLMRec combines LLM-based augmentation with noisy implicit-feedback pruning and MAE-based feature enhancement to refine augmented data.These mechanisms target the reliability problems associated with generated interactions and attributes.
  • Evaluation: Experiments on real-world benchmark datasets demonstrate superiority over state-of-the-art baseline methods while addressing recommendation accuracy and sparsity.The introduction also reports analysis and ablation studies examining the impact of LLM-enhanced augmentation strategies.

2 PRELIMINARY

The preliminary formulation separates collaborative graph embedding from preference prediction and then incorporates side information through a feature encoder. LLMRec extends this formulation by jointly using original and augmented edges and node features to produce preference scores.

  • Recommendation with Graph Embedding: Collaborative filtering learns user and item embeddings from sparse implicit-feedback edges for recommendation prediction.Graph neural networks model higher-order user-item relations in the interaction graph.
  • Recommendation with Side Information: Side-information recommenders introduce node features and optimize a feature graph encoder alongside collaborative embeddings.The encoder combines collaborative signals with side information in the final representation.
  • Recommendation with Data Augmentation: LLMRec augments the interaction graph with LLM-generated user-item edges and node features to address the risks of low-quality side information.The augmented graph contains both interaction augmentation and feature augmentation.
  • Recommendation with Data Augmentation: The recommender consumes original and augmented edges and features, producing representations used to rank user-item preference scores.The input consists of {E+, EA} and {F, FA}.

3 METHODOLOGY

The methodology is organized around four questions: generating user-item edges, producing valuable content, incorporating augmented content, and making the model robust to augmentation.

  • Methodological Questions: LLMRec’s methodology asks how LLMs can predict interactions, generate content, integrate it into the original graph, and robustify the model against augmented data.These four questions define the framework’s main methodological components.

3.1 LLMs as Implicit Feedback Augmentor (Q1)

LLMRec uses an LLM as a knowledge-aware sampler to generate pairwise user-item supervision from historical interactions, item content, and candidate items. Textual item representations support preference-aware selection, while sampling and pruning limit unreliable pseudo-labels and feedback noise.

  • LLMs as Implicit Feedback Augmentor: LLMRec samples pairwise positive and negative user-item training data from a natural-language representation of user preferences.The sampler is intended to increase effective supervision signals for sparse implicit feedback.
  • LLMs as Implicit Feedback Augmentor: For each user, the LLM receives historical interacted items with side information and selects likely or unlikely items from a candidate pool.The candidate pool is supplied because LLMs cannot rank all items, and candidates are drawn from hard samples proposed by a base recommender.
  • LLMs as Implicit Feedback Augmentor: Items are represented textually rather than by ID indexes so the recommender can use dataset content and more intuitively reflect user preferences.The textual format is part of the natural-language augmentation design.
  • LLMs as Implicit Feedback Augmentor: The augmented dataset adds LLM-selected positive and negative samples to the original training data for pairwise recommendation learning.The augmented edges comprise pairwise training triplets selected from candidate items using the user-interaction prompt.
  • LLMs as Implicit Feedback Augmentor: LLM-based sampling is intended to alleviate false-positive and false-negative issues in raw implicit feedback, while excessive pseudo-label inclusion can reduce accuracy.The method therefore samples only a subset of generated data and applies noise pruning.

3.2 LLM-based Side Information Augmentation

LLMRec augments recommender side information by using LLMs to generate user profiles and item attributes from interaction and content information. These augmented features are projected, combined with original representations, and normalized before recommendation.

  • User Profiling & Item Attribute Enhancing: LLMRec uses LLMs to generate user and item attributes that were not originally present in the dataset.The approach aims to summarize user profiles and produce informative, space-unified item attributes.
  • User Profiling & Item Attribute Enhancing: The side-information prompts P_Uu and P_Ii support attribute refinement for users and items, respectively.Generated textual attributes A_u and A_i are encoded as augmented features using the LLM embedding capability.
  • Side Information Incorporation: LLMRec incorporates augmented information through semantic projection, collaborative context injection, and feature incorporation.The projection uses linear layers with dropout to reduce dimensionality and map LLM-enhanced features into their own space.
  • Side Information Incorporation: Augmented features are added to ID embeddings with scale factors and normalization to adjust their influence in the recommender encoder.The resulting user and item representations combine original embeddings with LLM-augmented features.

3.3 Training with Denoised Robustification (Q4)

LLMRec trains on original and augmented interactions while constraining augmentation quality through noise pruning and MAE-based feature enhancement. These mechanisms reduce unreliable supervision and strengthen feature representations during optimization.

  • Augmented Optimization with Noise Pruning: LLMRec trains the recommender on the union of original interactions E and LLM-augmented interactions E_A.The objective uses BPR loss to exploit the additional supervisory signals from incorporated LLM-enhanced preferences.
  • Augmented Optimization with Noise Pruning: Only a sampled subset of LLM-generated data is added because excessive pseudo-label inclusion may degrade result accuracy.The augmented-set size is controlled by the batch size B and rate ω_3.
  • Augmented Optimization with Noise Pruning: Noise pruning sorts loss samples and retains N = (1 − ω_4) · |E ∪ E_A| examples to reduce unreliable gradient backpropagation.The procedure emphasizes relevant supervisory signals and is described as making optimization more stable and effective.
  • Enhancing Augmented Semantic Features via MAE: MAE-based feature enhancement masks a subset of node features and trains encoders through reconstruction objectives.Selected nodes receive a mask token, while unselected nodes retain their original augmented features.
  • Enhancing Augmented Semantic Features via MAE: The final optimization objective is a weighted sum of noise-pruned BPR loss and feature-restoration loss.The restoration loss compares masked augmented features with the original augmented feature matrix.

3.4 In-Depth Analysis of our LLMRec

The analysis identifies unreliable implicit feedback as a source of incorrect optimization directions and explains how LLMRec uses denoised, natural-language augmentation to improve training. It also reports complexity terms for the main components.

  • LLM-based Augmentation Facilitates Optimization: Non-interacted items can be false negatives, while interacted items can be false positives caused by accidental clicks or misleading titles.Mixing unreliable interactions with true preferences complicates accurate recommender construction.
  • LLM-based Augmentation Facilitates Optimization: Noisy interactions and non-interacted items can produce large optimization gradients pointing opposite to the true direction.For BPR, the cited gradients are ∇_u,i+ = 1 − σ(ŷ_u+−) and ∇_u,i− = σ(ŷ_u+−) − 1.
  • LLM-based Augmentation Facilitates Optimization: LLM-based augmentation uses natural-language space alongside ID vectors to reflect user preferences and reduce the impact of noisy and unobserved feedback.The conclusion states that this improves accuracy and speeds convergence.
  • LLM-based Augmentation Facilitates Optimization: The augmented semantic projection costs O(|U ∪ I| × d_LLM × d), while graph encoding costs O(L × |E+| × d).BPR computation costs O(d × |E ∪ E_A|), and feature reconstruction costs O(d × |Ṽ|).

4 EVALUATION

The evaluation tests LLMRec on Netflix and MovieLens using top-K recommendation metrics, compares it with diverse baselines, and examines component effects through ablations. Results report superior overall performance and show that augmentation and denoising components contribute to accuracy.

  • Datasets: Experiments use publicly available Netflix and MovieLens datasets with multimodal side information, including textual metadata and visual content.MovieLens includes titles, years, genres, and crawled movie posters.
  • Evaluation Protocol: Evaluation uses Recall, NDCG, and Precision at K values of 10, 20, and 50 under all-ranking, with results averaged over five independent runs.Statistical significance is assessed using p-values against the best-performing baseline.
  • Performance Comparison: LLMRec is compared against general collaborative-filtering, side-information, data-augmentation, and self-supervised recommendation baselines.The baseline groups include MF-BPR, NGCF, LightGCN, VBPR, MMGCN, GRCN, LATTICE, CLCRec, MMSSL, and MICRO.
  • Performance Comparison: LLMRec outperforms the baselines by augmenting user-item edges and improving side-information quality.The comparison uses LATTICE’s encoder with both an ID-corresponding encoder and a feature encoder.
  • Ablation Study: Ablations show that removing LLM-augmented implicit feedback, user profiling, or augmented user and item side information decreases recommendation accuracy.These findings support the contribution of interaction augmentation and LLM-enhanced side information.
  • Ablation Study: Removing noise pruning worsens performance, while removing both pruning and MAE-based feature enhancement also harms results.The ablations support the denoised data robustification mechanism.

4.4 Hyperparameter Analysis (RQ3)

Hyperparameter analysis examines how LLM generation settings, candidate limits, pruning, augmented-sample quantity, feature scaling, and augmentation reuse affect recommendation results and cost-effectiveness. The results emphasize constrained noise, moderate augmentation, and model-agnostic gains.

  • Parameters Affecting Augmented Data Quality: Increasing LLM temperature initially improves most metrics, followed by a decrease.The evaluated temperatures are 0, 0.6, 0.8, and 1.
  • Parameters Affecting Augmented Data Quality: Smaller top-p values tend to yield better results, while C = 10 produces the best results among the tested candidate counts.The candidate counts explored are 3, 10, and 30.
  • Parameters Affecting Augmented Data Quality: ω4 = 0 yields the worst result, highlighting the need to constrain noise in augmented implicit feedback.Prune rates from 0.0 to 0.8 are evaluated on both datasets.
  • Sensitivity of Recommenders to the Augmented Data: Suboptimal outcomes occur when augmented samples are zero or excessively large, and when feature scale ω2 is too small or too large.The study reports that greater diversity and randomness can support more robust gradient descent, while optimal feature scaling depends on the dataset.
  • Model-agnostic Property: LLM-based augmented data improves the performance of MICRO, MMSSL, and LATTICE on Netflix, demonstrating effectiveness and reusability across models.Some results do not surpass LLMRec, potentially because those baselines lack quality constraints and shared collaborative-signal modeling.
  • Cost/Improvement Conversion Rate: The user-profiling strategy is the most cost-effective augmentation strategy, and the overall investment is worthwhile.CIR compares improvement with LLM expenditure, including implicit-feedback and side-information augmentation costs.

5 RELATED WORK

Content-based recommenders incorporate auxiliary multimodal side knowledge to model modality-aware and higher-order collaborative signals. Recent methods align modal and collaborative signals through contrastive self-supervised learning.

  • VBPR combines traditional collaborative filtering with visual features, while MMGCN and GRCN use GNNs to capture modality-aware higher-order collaborative signals.
  • MMSSL and MICRO align modal signals with collaborative signals through contrastive self-supervised learning.This alignment reveals informative aspects of modal signals that benefit recommendations.

6 CONCLUSION

LLMRec addresses sparse implicit feedback and low-quality side information through LLM-enhanced user profiling, item-attribute debiasing, and denoised augmentation. The paper supports its effectiveness with theoretical analysis and benchmark experiments, while identifying causal and counterfactual extensions for future work.

  • LLMRec profiles user interaction preferences and debiases item attributes to address sparse implicit feedback and low-quality side information.
  • A denoised augmentation robustification mechanism is introduced to ensure the quality of augmented data.
  • Theoretical analysis and benchmark experiments support LLMRec’s effectiveness and superiority over state-of-the-art recommendation techniques.
  • Future work includes integrating causal inference into side-information debiasing and exploring counterfactual factors for context-aware user preference.
Loading 2311.00423v6…