PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Open-World Recommendation with Knowledge Augmentation from Large Language Models

Yunjia Xi, Weiwen Liu, Jianghao Lin, Xiaoling Cai, Hong Zhu, Jieming Zhu, Bo Chen, Ruiming Tang, Weinan Zhang, Rui Zhang, Yong Yu

arXiv:2306.10933v4cs.IR

TL;DR

Closed-domain recommender systems lack open-world knowledge, while direct LLM recommenders face accuracy, latency, and compositional limitations. KAR extracts user-preference reasoning and item facts, adapts them into recommendation-compatible vectors, and prestores them for efficient use, outperforming state-of-the-art methods and improving Huawei online A/B tests by 7% and 1.7%.

  • Problem

    Closed recommender systems have limited access to open-world knowledge, while direct LLM recommenders have accuracy, latency, and compositional-gap limitations.

  • Method

    KAR uses factorization prompting to obtain user-preference reasoning and item factual knowledge, transforms them with a hybrid-expert adaptor, and prestores the resulting vectors.

  • Results

    KAR significantly outperforms state-of-the-art methods, with 7% and 1.7% online A/B-test improvements on Huawei news and music recommendation platforms.

  • Takeaways & Limitations

    The augmented vectors can enhance any recommendation model while preprocessing and prestoring support fast training and inference.

  • Takeaways & Limitations

    Direct LLM recommendation remains constrained by accuracy, latency, and compositional-gap limitations in real-world settings.

Abstract

from arXiv · show

Recommender systems play a vital role in various online services. However, the insulated nature of training and deploying separately within a specific domain limits their access to open-world knowledge. Recently, the emergence of large language models (LLMs) has shown promise in bridging this gap by encoding extensive world knowledge and demonstrating reasoning capability. Nevertheless, previous attempts to directly use LLMs as recommenders have not achieved satisfactory results. In this work, we propose an Open-World Knowledge Augmented Recommendation Framework with Large Language Models, dubbed KAR, to acquire two types of external knowledge from LLMs -- the reasoning knowledge on user preferences and the factual knowledge on items. We introduce factorization prompting to elicit accurate reasoning on user preferences. The generated reasoning and factual knowledge are effectively transformed and condensed into augmented vectors by a hybrid-expert adaptor in order to be compatible with the recommendation task. The obtained vectors can then be directly used to enhance the performance of any recommendation model. We also ensure efficient inference by preprocessing and prestoring the knowledge from the LLM. Extensive experiments show that KAR significantly outperforms the state-of-the-art baselines and is compatible with a wide range of recommendation algorithms. We deploy KAR to Huawei's news and music recommendation platforms and gain a 7\% and 1.7\% improvement in the online A/B test, respectively.

1 INTRODUCTION

Classical recommender systems are insulated from open-world knowledge, while direct LLM recommenders face accuracy, latency, and compositional-gap limitations. KAR bridges these systems by extracting reasoning and factual knowledge from LLMs, adapting it into recommendation-compatible vectors, and enabling efficient use through preprocessing and prestorage.

  • Classical recommender systems are trained and deployed within closed domains, limiting access to external knowledge that could improve accuracy and generalization.
  • Open-world recommendation uses reasoning knowledge about user preferences and factual knowledge about items as complementary external information.
  • Directly using LLMs as recommenders is limited by inferior predictive accuracy, excessive inference latency, and difficulty solving compositional recommendation problems.
  • KAR uses factorization prompting, hybrid-expert adaptation, and classical recommender models to combine LLM knowledge with collaborative signals.
  • KAR knowledge can be preprocessed and prestored, and deployment produced 7% and 1.7% online A/B-test improvements on Huawei news and music platforms.

2 RELATED WORK

Related work includes direct use of pretrained language models for recommendation and auxiliary use within traditional recommender systems. KAR instead extracts reasoning and factual knowledge, adapts it into recommendation vectors, and prestores the result for efficient inference.

  • Direct LLM recommenders convert recommendation tasks or user profiles into prompts but generally fall behind state-of-the-art recommendation algorithms.
  • Prior studies mainly overlook inference latency and the compositional gap when applying LLMs to recommendation.
  • KAR uses factorization prompting to extract reasoning and factual knowledge, adapts it into recommendation vectors, and prestores it for fast inference.
  • Traditional recommender systems use pretrained language models to encode textual features or provide extra knowledge for user and item representations.

3 PRELIMINARIES

The recommendation task is formulated as binary click prediction over multi-field categorical data, while industrial systems divide recommendation into candidate generation, ranking, and reranking stages.

  • The dataset consists of feature-label pairs, where each label indicates no-click or click.
  • Recommendation models predict each sample’s click probability from its categorical features.
  • Industrial recommender systems use candidate generation, ranking, and reranking to narrow down relevant items.
  • Classical models are typically trained on specific recommendation datasets and therefore overlook potential benefits from open-world knowledge.

4 METHODOLOGY

The methodology section first presents an overview of the proposed framework, then explains the details of each component.

  • The section begins with an overview of the proposed Open-World Knowledge Augmented Recommendation Framework with Large Language Models.
  • The methodology then elaborates on the framework’s individual components.
  • The section is organized from a framework-level description to component-level details.

4.1 Overview

KAR extracts recommendation-relevant open-world knowledge from LLMs, adapts it into compact representations, and incorporates those representations into recommendation models. Knowledge generation and encoding occur during preprocessing, while adaptation and recommendation are jointly trained end to end.

  • KAR uses factorization prompting to extract recommendation-relevant knowledge from LLMs, then encodes and adapts it for recommendation.The framework obtains reasoning and factual knowledge before transforming it into representations suitable for recommendation.
  • The knowledge adaptation stage bridges LLM semantic representations and the recommendation space through a knowledge encoder and hybrid-expert adaptor.
  • Knowledge generation and encoding are conducted through preprocessing, while the hybrid-expert adaptor and recommendation model are jointly trained end to end.

4.2 Knowledge Reasoning and Generation

KAR uses factorization prompting and two complementary prompt types to obtain reasoning knowledge about users and factual knowledge about items. The design addresses compositional reasoning difficulties and aligns item knowledge with inferred preferences.

  • LLMs pose extraction challenges because user interests require multiple reasoning steps and generated factual knowledge may be correct but irrelevant to inferred preferences.The latter mismatch can occur when user preferences concern awards while item knowledge focuses on storyline.
  • Factorization prompting decomposes complex preference reasoning into simpler subproblems for major scenario-specific factors, alleviating the compositional gap.Factors are identified through collaboration with LLMs and expert refinement.
  • KAR designs preference reasoning prompts and item factual prompts, using user context and scenario-specific factors to guide LLM outputs.Preference prompts use profile descriptions, behavior histories, and scenario factors; item prompts use item descriptions and the same factors.
  • Figure 3 distinguishes prompt templates, filled content, and LLM responses using green, purple, and yellow text bubbles, respectively.
  • The combined prompts make the LLM both a preference reasoner and a knowledge provider for extracting open-world recommendation knowledge.

4.3 Knowledge Adaptation

KAR converts textual LLM knowledge into compact recommendation-compatible vectors through a knowledge encoder and a hybrid-expert adaptor. Shared and dedicated experts fuse common and knowledge-type-specific information while reducing dimensionality and mitigating noise.

  • LLM-generated text cannot directly serve traditional recommenders, while large dense vectors occupy a mismatched semantic space.
  • 4.3.2 Hybrid-expert Adaptor: The hybrid-expert adaptor transforms aggregated semantic representations into the recommendation space, addressing dimensionality mismatch and reducing noise.
  • 4.3.1 Knowledge Encoder: The knowledge encoder produces aggregated dense vectors from token-level encodings of textual reasoning and factual knowledge.Average pooling is primarily used in practice, although [CLS] representations and other aggregation functions are possible.
  • 4.3.2 Hybrid-expert Adaptor: Shared experts capture information common to preference and factual representations, while dedicated experts model their unique characteristics.
  • 4.3.2 Hybrid-expert Adaptor: Expert outputs produce augmented vectors of size q, where q is much smaller than the original dimension m.

4.4 Knowledge Utilization

KAR incorporates reasoning and factual augmented vectors as additional input features in backbone recommendation models. Because it modifies only the backbone input, the framework is independent of the backbone design and can extend across recommendation tasks.

  • KAR directly treats reasoning and factual augmented vectors as additional feature fields that interact explicitly with existing recommendation features.
  • KAR modifies only the backbone input, making it independent of the backbone model’s design and loss function.
  • Adding two augmented vectors extends KAR to tasks such as sequential recommendation and direct recommendation.

4.5 Speed-up Approach

KAR accelerates inference by prestoring LLM-generated knowledge representations and, when needed, compact augmented vectors. This avoids repeated LLM inference during backbone-model training and serving while preserving near-backbone inference time.

  • KAR prestores LLM-generated knowledge representations in a database, using the LLM and knowledge encoder only before backbone-model training.During backbone training and inference, relevant representations are retrieved from storage.
  • Detaching the adaptor after training enables prestorage of smaller augmented vectors for stricter inference-time or storage requirements.The augmented vectors can be much smaller than the original knowledge representations, such as 32 versus 4096 dimensions.
  • Prestored augmented vectors reduce inference time to nearly that of the original backbone model and improve storage efficiency.The paper reports this strategy as an alternative to repeated online LLM computation.
  • Long-term user preferences are treated as sufficiently stable for prestorage, while recent behavior remains modeled by the backbone recommender.This design avoids requiring LLM access to real-time user behaviors.

5 EXPERIMENT

The experiments evaluate KAR across datasets, tasks, backbone models, knowledge sources, and industrial deployments. KAR improves recommendation performance broadly, with especially strong gains from preference reasoning and online tests.

  • CTR prediction: KAR improves all nine selected CTR backbones across two datasets, with approximately 1–1.5% AUC gains.On MovieLens-1M with FiBiNet, KAR increases AUC by 1.49% and decreases LogLoss by 2.27%.
  • Reranking: KAR increases PRM reranking performance on Amazon-Books by 5.71% in MAP@7 and 4.71% in NDCG@7.Gains are more pronounced for methods without history modeling, whereas MIR shows a slightly smaller enhancement.
  • Comparison with baselines: KAR outperforms pretrained-language-model baselines; with DIN on Amazon-Books, it improves AUC by 0.91% and LogLoss by 1.27% over TALLRec.Smaller PLM-based methods often provide modest gains, while LLM-based approaches produce more substantial improvements.
  • Other knowledge sources: LLM-derived knowledge improves backbones more than knowledge-graph enhancement, while combining both sources does not significantly exceed LLM knowledge alone.The authors attribute this partly to LLM coverage of typical knowledge-graph information and its additional user-preference reasoning.
  • Online deployment: KAR improves Huawei news Recall by 7% and music engagement metrics by 1.7% song play count, 1.64% playback devices, and 1.57% total duration.These results come from online A/B tests on the news and music platforms.
  • Knowledge ablation: Reasoning and factual knowledge each improve backbone performance, but reasoning knowledge contributes more; their combination produces a reported synergistic effect.The authors explain that factual item knowledge helps align externally generated preference reasoning with candidate items.
  • Encoder and adaptor analysis: ChatGLM-based encoders outperform BERT-based variants, and the hybrid-expert adaptor outperforms simpler MLP and MoE transformations with ChatGLM.The results indicate that both knowledge encoding and semantic-to-recommendation transformation affect KAR performance.
  • Efficiency: Direct LLM inference is unsuitable for recommender latency requirements because API responses take 4–6 seconds versus a typical 100-millisecond requirement.Even a smaller fine-tuned model such as TALLRec remains close to one second in inference latency.

6 BROADER IMPACT

KAR addresses privacy and security concerns in LLM-augmented recommendation by avoiding LLM fine-tuning and direct display of generated content. It converts LLM outputs into representations integrated with traditional recommender systems.

  • LLM-based recommendation requires explicit attention to privacy and security because these models may compromise user privacy or generate harmful content.
  • KAR does not fine-tune LLMs, helping prevent retention of user-specific data and privacy leakage through external APIs in real-world deployments.The framework can use in-house models such as Huawei’s PanGu for online experiments.
  • KAR converts LLM-generated text into robust representations before incorporating it into traditional recommender systems, rather than displaying the content directly to users.This design avoids exposing harmful or misleading generated content while permitting filtering mechanisms.

7 CONCLUSION

KAR incorporates open-world knowledge from LLMs into recommender systems through preference reasoning, item facts, and adaptable augmented vectors. The framework supports diverse recommendation models and efficient inference through preprocessing and prestoring knowledge.

  • KAR acquires reasoning knowledge about user preferences and factual knowledge about items from LLMs using factorization prompting.
  • A hybrid-expert adaptor transforms LLM-derived knowledge into augmented vectors compatible with recommendation tasks and models.
  • Preprocessing and prestoring LLM knowledge enables efficient inference, while the resulting vectors can enhance various recommendation algorithms.
Loading 2306.10933v4…