PaperlayerView exact revision on arXiv

Source-linked AI summary

CoRAL: Collaborative Retrieval-Augmented Large Language Models Improve Long-tail Recommendation

Junda Wu, Cheng-Chun Chang, Tong Yu, Zhankui He, Jianing Wang, Yupeng Hou, Julian McAuley

arXiv:2403.06447v1cs.IRcs.AI

TL;DR

Long-tail recommendation is difficult with sparse, imbalanced interaction data, and semantic-only LLM reasoning can miss collaborative patterns. CoRAL retrieves user-item interactions and learns a reinforcement-learning policy to select minimal-sufficient prompt evidence. The paper reports average improvements of up to 21.1% in AUC and 25.1% in F1.

  • Problem

    Long-tail recommendation faces sparse, imbalanced interactions, while LLMs relying on item semantics can lack collaborative evidence for task-specific user-item patterns.

    Source: §1 · pp. 1–2

  • Method

    CoRAL retrieves collaborative interactions into LLM prompts and learns a reinforcement-learning policy to select minimal-sufficient evidence for each recommendation.

    Source: §1 · p. 2

  • Results

    CoRAL reports average improvements of up to 21.1% in AUC and 25.1% in F1 as it aligns LLM reasoning with user-item interaction patterns.

    Source: §5.2.2 · p. 7

  • Takeaways & Limitations

    A warm start from popular-item collaborative-filtering representations can make exploration in continuous embedding space more efficient.

    Source: §5.3.2 · p. 8

Abstract

from arXiv · show

The long-tail recommendation is a challenging task for traditional recommender systems, due to data sparsity and data imbalance issues. The recent development of large language models (LLMs) has shown their abilities in complex reasoning, which can help to deduce users' preferences based on very few previous interactions. However, since most LLM-based systems rely on items' semantic meaning as the sole evidence for reasoning, the collaborative information of user-item interactions is neglected, which can cause the LLM's reasoning to be misaligned with task-specific collaborative information of the dataset. To further align LLMs' reasoning to task-specific user-item interaction knowledge, we introduce collaborative retrieval-augmented LLMs, CoRAL, which directly incorporate collaborative evidence into the prompts. Based on the retrieved user-item interactions, the LLM can analyze shared and distinct preferences among users, and summarize the patterns indicating which types of users would be attracted by certain items. The retrieved collaborative evidence prompts the LLM to align its reasoning with the user-item interaction patterns in the dataset. However, since the capacity of the input prompt is limited, finding the minimally-sufficient collaborative information for recommendation tasks can be challenging. We propose to find the optimal interaction set through a sequential decision-making process and develop a retrieval policy learned through a reinforcement learning (RL) framework, CoRAL. Our experimental results show that CoRAL can significantly improve LLMs' reasoning abilities on specific recommendation tasks. Our analysis also reveals that CoRAL can more efficiently explore collaborative information through reinforcement learning.

1 INTRODUCTION

Long-tail recommendation is challenged by sparse, imbalanced interactions, while LLM reasoning based mainly on item semantics can miss collaborative user-item patterns. CoRAL retrieves collaborative evidence and learns a policy to select minimal-sufficient support for its prompts.

1 INTRODUCTION

  • Sparse, imbalanced interactions make long-tail patterns difficult to learn, while trained models can overfit popular items amid selection, exposure, and popularity bias.These challenges motivate recommendations that account for user preferences beyond popular-item patterns.Source: §1 · p. 1
  • Data rebalancing can be suboptimal because learning from long-tail data is inefficient, and the resulting compromise can also cause knowledge forgetting on popular items.With few interactions for long-tail items, fine-grained reasoning about user preferences becomes important.Source: §1 · pp. 1–2
  • LLMs can reason over item semantics, but semantic similarity alone may misalign recommendations with real user-item interaction patterns.A semantically similar item may be recommended even when collaborative evidence would argue against it.Source: §1 · p. 2, Figure 1 · p. 2
  • CoRAL retrieves user-item interactions as collaborative prompt evidence and uses reinforcement learning to select minimal-sufficient information for each recommendation task.The retrieval policy balances exploration of users and items with exploitation of collected information to improve prediction accuracy.Source: §1 · p. 2
  • CoRAL initializes its retrieval policy with user and item representations learned from popular-item data to improve data efficiency and reduce early exploration time.The warm start is intended to help address sparse rewards during policy learning.Source: §1 · pp. 2–3

2 RELATED WORKS

Prior long-tail recommendation work includes transfer learning and causal debiasing, while LLM recommenders use semantic content and collaborative prompting. CoRAL distinguishes itself with lightweight collaborative retrieval for long-tail settings.

2.1 Long-tail Recommendation

  • Long-tail methods include transfer approaches linking head and tail items [70] [72], sequential domain transfer, and causal-debiasing frameworks [6] [75].The paper notes that popularity bias is less discussed in the cited transfer designs and proposes LLM reasoning over collaborative information.Source: §2.1 · p. 3

2.2 LLMs in Recommender Systems

  • LLM recommenders use item embeddings [16], content augmentation [4] [57] [62] [69], personalized reasoning [53], and collaborative reranking [54], but interaction-pattern misalignment remains.The paper attributes this misalignment to insufficient collaborative information.Source: §2.2 · p. 3
  • Soft-prompt alignment [71] [74] and continuous prompt vectors [23] inject collaborative knowledge, but their larger data needs are unsuitable for long-tail settings; CoRAL instead uses lightweight retrieval.Source: §2.2 · p. 3

3 PROBLEM FORMULATION

The paper formulates long-tail recommendation as predicting a user's preference for a long-tail item with LLM reasoning supported by retrieved interactions. It models evidence retrieval as a sequential decision process whose policy seeks minimal-sufficient support through cumulative reward.

3 PROBLEM FORMULATION

  • The task is to predict a user's preference for a long-tail item in a collaborative-filtering recommender, framing long-tail recommendation as an LLM reasoning problem.Source: §3 · p. 3
  • For a user-item pair, the system combines the user's prior interactions with sequences of supporting users and items retrieved as collaborative evidence.The retrieval policy selects the next user-item pair at each step because prompt context is limited, with the goal of finding minimal-sufficient support.Source: §3 · p. 3

3.1 MDP Formulation for Retrieval Policy

  • The retrieval process is an MDP with a continuous state and action space, state transitions, and a reward function.Source: §3.1 · p. 3
  • The state encodes collaborative information from retrieved users and items, including their collaborative preference patterns.Source: §3.1 · p. 3
  • At each step, the action generates queries for the next user and item, seeking relevant candidates while exploring under-explored regions.Source: §3.1 · p. 3
  • The policy maximizes cumulative discounted reward to obtain minimal-sufficient information support for the LLM.The discount rate weights future rewards in the objective.Source: §3.1 · p. 3

3.2 Reward Function

  • At each retrieval step, the LLM predicts the user's rating from a prompt containing the user's history and the collaborative information collected so far.Source: §3.2 · p. 4
  • The reward represents marginal information gain measured by prediction discrepancy, favoring retrieved user-item pairs that lead to more accurate LLM predictions.The ground-truth preference label comes from the training rating matrix.Source: §3.2 · p. 4

4 PROPOSED FRAMEWORK: CORAL

CoRAL combines LLM recommendation prompting with collaborative interaction evidence and learns a retrieval policy to select user-item context for the prompt.

4.1 Collaborative Prompting

  • The prompt combines collaborative evidence about users’ shared item preferences with the target user’s previously interacted items and their descriptions.For each retrieved item, the prompt summarizes which users like or dislike it and asks the LLM to identify the preference pattern.Source: §4.1 · p. 4

4.2 Retrieval Policy Network

  • The retrieval policy seeks a minimal-sufficient prompt by maximizing long-term information gain under limited prompt capacity.The LLM is frozen while a lightweight retrieval policy is learned.Source: §4 · p. 4, §4.2 · p. 5
  • The policy learns continuous user-item embedding actions with DDPG, a low-rank decision space that supports scaling to new users and items at inference.Source: §4.2 · p. 5, §4.2.2 · p. 5

4.2.1 State Representation.

  • Popular-item embeddings warm-start the retrieval policy to improve exploration efficiency and address sparse rewards.Random early exploration can retrieve similar users and items, making rewards sparse.Source: §4.2.1 · p. 5

4.2.2 User-item Retrieval.

  • For each target pair, the policy queries the embedding space and retrieves the nearest user and item by Euclidean distance for the collaborative sets.Source: §4.2.2 · p. 5

4.2.3 State Transition.

  • An MLP models state transitions to track retrieval and aggregate collected collaborative information.Source: §4.2.3 · p. 5

4.3 Minimal-sufficient Collaborative Information via Reinforcement Learning

  • The retrieval policy uses an actor-critic DDPG framework, with critic-based policy gradients and exploration noise to seek informative users and items.The target actor is synchronized after each update; the exploration process uses Ornstein–Uhlenbeck noise.Source: §4.3 · p. 5

5 EXPERIMENTS

The experiments examine collaborative evidence, evidence sufficiency, and prompt-size efficiency across four Amazon Product tasks.

5 EXPERIMENTS

  • The experiments ask whether collaborative information aligns LLM reasoning with recommendation tasks and whether CoRAL finds sufficient, minimally sufficient evidence.Source: §5 · p. 5

5.1 Experimental Settings

  • Evaluation covers four Amazon Product tasks; Appliances and Gift Cards are described with review, user, and product counts.Appliances has 602,777 reviews, 515,650 users, and 30,252 products; Gift Cards has 147,194 reviews, 128,877 users, and 1,548 products.Source: §5.1.1 · p. 5

5.1.1 Datasets.

  • The training procedure takes episode length and maximum episode steps as inputs and initializes actor and critic networks.Source: §5.1.1 · p. 6
  • The procedure constructs a user-preference prompt and obtains an initial prediction before iterating through the episode.Source: §5.1.1 · p. 6

User-item Retrieval

The retrieval step uses the policy’s current action to select a user-item pair for the collaborative support sets.

User-item Retrieval

Collaborative Prompting

The appendix specifies dataset handling, evaluation measures, comparison methods, and implementation settings for the experiments.

Collaborative Prompting

  • Prime Pantry and Software goods are Amazon subsets whose item descriptions come from metadata descriptions and product titles, respectively.Prime Pantry contains 471,614 reviews, 247,659 users, and 10,814 products; Software goods contains 459,436 reviews, 375,147 users, and 21,663 products.Source: Collaborative Prompting · p. 6
  • Preprocessing uses the 80/20 popularity rule for long-tail items, filters users and items below five interactions, and reserves 30% of long-tail data for validation and testing.The remaining 70% of long-tail data and the rest of the dataset are used for training; five-score ratings are binarized at threshold 3.Source: Collaborative Prompting · p. 6

5.1.2 Metrics.

  • AUC and F1 are the reported evaluation metrics for long-tail recommendation and collaborative filtering.Source: §5.1.2 · p. 6

5.1.3 Baselines.

  • The comparison set spans collaborative-filtering and popularity-debiasing baselines, plus LLM-Language, which prompts from interacted items and user preferences.The listed collaborative-filtering methods include AFM, DCN, DFM, and WDL; the debiasing methods include IPS and CausE.Source: §5.1.3 · p. 6
  • CoRAL variants test policy initialization and random retrieval, alongside the experimental results table reporting AUC and F1 on four Amazon datasets.CoRAL-Method initializes the retrieval policy with a method; CoRAL-random retrieves items randomly.Source: §5.1.3 · pp. 6–7, Table 1 · p. 7
  • The retrieval policy uses DDPG with a 1,000-entry buffer, batch size 16, 128-dimensional user and item actions, and up to 2,000 learning iterations.The setup uses Adam at learning rate 0.001 and Ornstein–Uhlenbeck exploration noise with standard deviation 0.1.Source: §5.1.4 · p. 7

5.1.4 Implementation Details.

  • GPT-4 supplies the reinforcement-learning reward; training allows up to 10 interactions per episode, while evaluation retrieves five rounds.Training can stop early when the absolute prediction–rating discrepancy is below 0.1.Source: §5.1.4 · p. 7

5.2 Recommendation Performance (RQ1)

CoRAL adds collaborative evidence to LLM prompts and uses a learned retrieval policy to align reasoning with recommendation-task interaction patterns. Its reported average gains are substantial, although random retrieval produces different outcomes across AUC and F1.

5.2.1 Effect of the Retrieval Policy.

  • CoRAL-random consistently outperforms LLM-Language in AUC, but generally performs worse in F1 except on Prime Pastry.The authors suggest that uncurated, irrelevant information may add bias and impair the precision-recall trade-off.Source: §5.2.1 · p. 7
  • CoRAL improves AUC and F1 scores by up to 21.1% and 25.1%, respectively, on average.The passage attributes these gains to aligning LLM reasoning with user-item interaction patterns through reinforcement learning.Source: §5.2.2 · p. 7
  • LLM-based methods sometimes perform substantially worse than traditional recommendation baselines, including on Prime Pastry and Appliances.The authors describe this inconsistency as evidence of misalignment between LLM reasoning and specific recommendation tasks.Source: §5.2.2 · p. 7

5.3 Sufficient Collaborative Information from Popular Items (RQ2)

The analysis examines whether popular-item data can help CoRAL’s retrieval policy learn more efficiently. Popular-item initialization generally outperforms random initialization, while the learning curves reveal a trade-off between actor convergence and critic loss.

5.3.1 Comparison to Randomly Initialized Policy.

  • Policies initialized using models learned from popular items generally perform better than randomly initialized policies.The authors identify this as a data-efficiency advantage of popular-item initialization.Source: §5.3.1 · p. 7
  • Randomly initialized policies’ actors converge faster, but their critics have higher learning loss than policies with short-head initialization.The comparison uses actor and critic learning curves on Gift Cards and Prime Pantry, selected as challenging datasets.Source: §5.3.2 · p. 8, Figure 2 · p. 8

5.3.2 Actor-critic Learning Curves.

  • Pre-trained user and item embedding spaces from the short-head training data make exploration in the continuous embedding space more efficient.The experiments use DFM and WDL as backbones to examine learning from popular items.Source: §5.3 · p. 7, §5.3.2 · p. 8

5.4 Minimally-sufficient Collaborative Information from Iterative Retrieval (RQ3)

The iterative-retrieval analysis reports information gains across rounds, with a decreasing gain margin. It also contrasts the exploration patterns and final performance of the DFM and DCN backbones.

5.4 Minimally-sufficient Collaborative Information from Iterative Retrieval (RQ3)

  • Across CoRAL policies, each retrieval iteration brings informative users and items and consistent information gain, though the gain margin decreases.The passage describes model performance with respect to retrieval rounds.Source: §5.4 · p. 8
  • The cited ablation compares CoRAL performance with and without short-head data initialization for DFM and WDL backbones.The caption identifies the comparison scope but does not provide the outcome.Source: Table 2 · p. 9
  • DCN behaves more greedily and reaches upper-bound performance sooner, while DFM explores more early and achieves better final performance.The authors point to this contrast as evidence of the importance of the exploration-exploitation trade-off.Source: §5.4 · pp. 8–9

6 CONCLUSION

The conclusion presents CoRAL as a collaborative retrieval-augmented approach for long-tail collaborative filtering. It uses reinforcement learning to identify a minimal, sufficient set of user-item interactions as evidence for LLM reasoning.

6 CONCLUSION

  • CoRAL targets long-tail items in collaborative filtering, where data sparsity and imbalance hamper recommendation.The conclusion frames the method for collaborative filtering-based recommender systems.Source: §6 · p. 9
  • CoRAL retrieves collaborative evidence to align LLM reasoning with user-item interaction patterns rather than relying only on item semantics.The retrieved interactions serve as supporting evidence for deducing user preferences and interaction dynamics.Source: §6 · p. 9
  • A reinforcement-learning retrieval policy identifies an optimal, minimally sufficient set of user-item interactions for the LLM’s reasoning.The conclusion reports this strategy as improving LLM-based recommendation.Source: §6 · p. 9
Loading 2403.06447v1…