PaperlayerView exact revision on arXiv

Source-linked AI summary

Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approach

Junjie Zhang, Ruobing Xie, Yupeng Hou, Wayne Xin Zhao, Leyu Lin, Ji-Rong Wen

arXiv:2305.07001v1cs.IRcs.CL

TL;DR

Existing recommenders mainly learn from historical interactions and have limited generalization to unseen users or tasks. InstructRec treats recommendation as natural-language instruction following, using an instruction-tuned open-source LLM and diverse personalized instruction data. Experiments report effectiveness and generalization across recommendation and search scenarios, while future work targets larger models, longer contexts, and multi-turn interaction.

  • Problem

    Existing recommendation algorithms fit user-item interaction data but lack generalization to unseen users or tasks, while recommendation needs accurate understanding of varied user needs.

  • Method

    InstructRec instruction-tunes a 3B Flan-T5-XL using a format that integrates preference, intention, task form, and context, with automatically generated personalized instruction data.

  • Results

    Extensive experiments report effectiveness and generalization across diverse recommendation scenarios, including held-out instructions and domains.

  • Takeaways & Limitations

    Recommendation can be formulated as instruction following, allowing users to express diverse information needs in natural language.

  • Takeaways & Limitations

    Future work will scale the LLM, extend context length for long behavior sequences, and support multi-turn chit-chat interaction.

Abstract

from arXiv · show

In the past decades, recommender systems have attracted much attention in both research and industry communities, and a large number of studies have been devoted to developing effective recommendation models. Basically speaking, these models mainly learn the underlying user preference from historical behavior data, and then estimate the user-item matching relationships for recommendations. Inspired by the recent progress on large language models (LLMs), we take a different approach to developing the recommendation models, considering recommendation as instruction following by LLMs. The key idea is that the preferences or needs of a user can be expressed in natural language descriptions (called instructions), so that LLMs can understand and further execute the instruction for fulfilling the recommendation task. Instead of using public APIs of LLMs, we instruction tune an open-source LLM (3B Flan-T5-XL), in order to better adapt LLMs to recommender systems. For this purpose, we first design a general instruction format for describing the preference, intention, task form and context of a user in natural language. Then we manually design 39 instruction templates and automatically generate a large amount of user-personalized instruction data (252K instructions) with varying types of preferences and intentions. To demonstrate the effectiveness of our approach, we instantiate the instruction templates into several widely-studied recommendation (or search) tasks, and conduct extensive experiments on these tasks with real-world datasets. Experiment results show that the proposed approach can outperform several competitive baselines, including the powerful GPT-3.5, on these evaluation tasks. Our approach sheds light on developing more user-friendly recommender systems, in which users can freely communicate with the system and obtain more accurate recommendations via natural language instructions.

1 INTRODUCTION

Existing recommender systems primarily fit historical interactions and struggle to generalize to unseen users or tasks. InstructRec instead treats recommendation as natural-language instruction following and adapts an LLM through instruction tuning.

  • Historical-interaction recommenders lack generalization to unseen settings such as new users or new tasks.
  • Recommendation requires natural-language instructions that capture user needs, interaction history, preferences or intentions, and other personalized factors.
  • LLMs require recommendation-specific tuning to interpret behavior data and solve complex specialized recommendation tasks.
  • InstructRec lets users express specific needs in natural language and treats recommendation as instruction following by an LLM.
  • The approach combines preference, intention, task form, and context in a general instruction format, with 39 templates and 252K automatically generated personalized instructions.
  • Experiments across real-world interaction scenarios show effectiveness and improved generalization on diverse user needs, held-out instructions, and held-out domains.

2 METHODOLOGY

The methodology develops InstructRec as a natural-language instruction framework for recommendation. It defines instruction components, constructs diverse instruction data, and fine-tunes a language model for user-centric recommendation.

  • InstructRec allows users to express information needs in natural-language instructions during recommender-system interaction.
  • The approach uses a 3B Flan-T5-XL base model, while noting that the minimum size distinguishing small language models from LLMs lacks consensus.
  • The method designs a recommendation instruction format and instantiates it across interaction scenarios using preference, intention, task form, and context.

2.1 Instruction Format for Recommendation

The instruction format represents user preferences, immediate intentions, task forms, and contextual information, enabling multiple recommendation and search scenarios. Its main practical focus is reranking, while instruction diversity supports evaluation.

  • 2.1 Instruction Format for Recommendation: The format combines preference, intention, task form, and optional context to reveal user needs and clarify recommendation settings.
  • 2.1.1 Key Aspects in Instructions.: Preference captures long-term personalized tastes and is categorized as none, implicit, or explicit preference.
  • 2.1.1 Key Aspects in Instructions.: Intention represents immediate demands distinct from long-term preferences and ranges from none to vague or specific intention.
  • 2.1.1 Key Aspects in Instructions.: Task forms include pointwise recommendation, pairwise recommendation, matching for candidate generation, and reranking of retrieved candidates.
  • 2.1.2 Instantiation for Various Interaction Scenarios.: Instruction examples are organized in Table 1 by preference, intention, and task form, with modifications made for readability.
  • 2.1.2 Instantiation for Various Interaction Scenarios.: Preference-only instructions model traditional recommendation, while preference-free instructions process vague or specific queries as retrieval.
  • 2.1.2 Instantiation for Various Interaction Scenarios.: Combining preferences with intentions supports personalized search by reranking candidate items according to user needs.
  • 2.1.2 Instantiation for Various Interaction Scenarios.: The framework treats recommendation and search similarly, focuses mainly on reranking because inference is costly, and finds instruction diversity effective in evaluation.

2.2 Instruction Generation

The paper generates recommendation instructions by combining manually designed templates with personalized preferences and intentions derived from interaction data or teacher-LLM outputs. It expands instruction diversity through task reversal, preference–intention alignment, and chain-of-thought reasoning.

  • 2.2 Instruction Generation: Instruction generation addresses the difficulty of obtaining large-scale data that explicitly reveals users’ preferences or intentions.The approach simulates these aspects from available interaction data and uses an instruction-tuned teacher-LLM for automatic generation.
  • 2.2 Instruction Generation: The pipeline first creates coarse-grained templates for interaction scenarios, then fills them with fine-grained preferences and intentions.Fine-grained information is extracted from interaction data or generated by the teacher-LLM.
  • 2.2.1 Annotating the Aspects in Instructions: Implicit preferences use item titles and historical interactions, while explicit preferences are generated by GPT-3.5 from users’ behavioral histories.The generated expressions are intended to reflect users’ real preferences based on target-item information and prior behavior.
  • 2.2.1 Annotating the Aspects in Instructions: Vague intentions are extracted from target reviews, whereas specific intentions are derived by concatenating target-item category labels.The two forms differ in clarity, with category information providing a more specific representation of intended items.
  • 2.2.1 Annotating the Aspects in Instructions: The main task forms are pointwise recommendation, next-item matching, and candidate-list reranking.Their responses respectively use yes/no prediction, next-item prediction, or selection from provided candidates.
  • 2.2.2 Increasing the Diversity of Instructions: Instruction diversity is increased by reversing normal input-output roles and enforcing relatedness between short-term intentions and long-term preferences.These strategies train the model to infer information needs from recommendation feedback and connect behavioral evidence with intended requests.
  • 2.2.2 Increasing the Diversity of Instructions: Chain-of-thought-like instructions add intermediate reasoning from implicit behavior to explicit preferences or intentions before recommendation.The desired response records inferred preferences and then recommends a target item.
  • 2.2.2 Increasing the Diversity of Instructions: The authors manually design 39 coarse-grained templates and generate 252K fine-grained personalized instructions with GPT-3.5.The generated instruction set is summarized in Table 2, while quality evaluation is reported in Table 3.

2.3 Instruction Tuning for Recommendations

InstructRec adapts a language model to recommendation by instruction tuning on natural-language instructions and target responses. At inference, it uses the tuned model as a reranker over candidate items selected from user instructions and historical information.

  • 2.3 Instruction Tuning for Recommendations: The instruction-tuning procedure optimizes the backbone LLM using generated recommendation-oriented instruction data.The paper presents the backbone, optimization, and inference stages for adapting the model to recommender systems.
  • 2.3 Instruction Tuning for Recommendations: InstructRec treats recommendation as a user-centric instruction-following task implemented with a large language model.The approach aims to follow instructions that specify users’ needs across interaction scenarios.
  • 2.3.2 Training and Inference.: Target outputs are supervised responses matched to instruction types, such as the next item or a user’s reasoning process.This makes different recommendation tasks trainable within the same response-generation framework.
  • 2.3.2 Training and Inference.: Natural-language instructions and target outputs unify training as a sequence-to-sequence problem.Both sides of the supervised examples can be formatted as natural language.
  • 2.3.2 Training and Inference.: The training objective is the negative log-likelihood of desired system responses conditioned on instructions.Y_k denotes the desired response for instance k, I_k the corresponding instruction, and B the batch size.
  • 2.3.2 Training and Inference.: Table 4 compares InstructRec with two related studies in terms of instruction tuning and recommendation-oriented design.The table’s comparison is framed around the proposed approach and related instruction-based methods.
  • 2.3.2 Training and Inference.: At inference, InstructRec acts as a reranker that produces the final ranking of candidates using users’ instructions.The system selects coarse-grained templates using user-issued instructions and other information such as historical interactions.

2.4 Discussion

The discussion contrasts InstructRec with traditional recommenders and prior LLM-based systems. Its distinguishing focus is aligning recommendation instructions with diverse, detailed user needs in practical interactions.

  • 2.4 Discussion: Traditional recommenders commonly use user and item identifiers with specialized preference functions, but may struggle with cold-start users or items.They also emphasize passive acceptance of recommendations, which may not fully capture users’ preferences.
  • 2.4 Discussion: Prior LLM recommender systems formulate behavior modeling as language modeling and incorporate contextual information into sequential text.P5 and M6-Rec are cited as examples of this direction.
  • 2.4 Discussion: InstructRec differs by designing recommendation instructions with diverse preferences, intentions, and task forms rather than mainly task-specific formulations.This design targets greater personalization across interaction scenarios.

3 EXPERIMENTS

Experiments evaluate InstructRec across sequential recommendation, product search, and personalized search using varied user information needs. InstructRec generally outperforms or matches competitive baselines, with instruction tuning helping accommodate personalized and ambiguous requests.

  • Experimental Setup: Experiments cover sequential recommendation, product search, and personalized search with implicit, explicit, vague, and specific user information.Evaluation uses real-world datasets, top-K HR and NDCG, and scenario-specific baselines.
  • Sequential Recommendation: InstructRec outperforms sequential-recommendation baselines despite the challenge of representing users’ implicit behavioral sequences.The authors attribute this setting’s difficulty to the specificity and complexity of private-domain behavior data.
  • Product Search: InstructRec achieves superior or comparable performance in most product-search cases, especially on top-ranking metrics such as NDCG@1.Product-search instructions are relatively specific, while test items are unseen during training.
  • Personalized Search: InstructRec outperforms other approaches by a large margin in almost all personalized-search cases.Traditional models perform poorly with ambiguous instructions, while instruction tuning helps align the universal LLM with personalized behaviors.
  • Overall Findings: Across classical tasks, InstructRec achieves the best performance in almost all cases regardless of implicit or explicit preferences and vague or specific intentions.The overall result spans sequential recommendation, product search, and personalized search.

3.3 Further Analyses

Further analyses examine reranking harder and larger candidate sets, instruction diversity, and transfer to unseen datasets. Results show strong in-domain generalization, while zero-shot transfer remains below in-domain-trained traditional models.

  • Reranking Hard Candidates: InstructRec is evaluated in a matching-then-reranking pipeline using candidates retrieved by a separate matching module.The setup is intended to approximate practical recommendation systems more closely than reranking randomly sampled candidates.
  • Discriminating More Candidate Items: InstructRec has a considerable performance advantage over the traditional baseline when discriminating one hundred candidate items.The authors view reranking as a suitable deployment stage because it is closest to user communication.
  • Discriminating More Candidate Items: HR@1 improves less than other metrics in the larger-candidate setting, motivating longer contexts and more efficient algorithms for handling many candidates.The authors attribute this pattern potentially to the difficulty of distinguishing among selected items under limited context length.
  • Effects of Instructions: Performance steadily improves on the held-out personalized-search scenario as more interaction scenarios are included during instruction tuning.The held-out scenario uses personalized search with vague intention.
  • Effects of Instructions: Adding the “vague intention∗” instruction produces a large performance improvement on the held-out scenario.The authors describe this result as motivation for annotating more diverse data through self-instruction strategies.
  • Generalization across Datasets: On an unseen dataset, InstructRec’s zero-shot performance falls short of traditional sequential models trained on in-domain instances.Nevertheless, it outperforms other powerful LLMs on the transfer setting.

4 CONCLUSION AND FUTURE WORK

The paper concludes that InstructRec treats recommendation as instruction following, using natural-language user needs and instruction-tuned LLMs. Future work targets larger models, longer behavioral contexts, and multi-turn interaction.

  • Conclusion: InstructRec lets users express information needs as natural-language instructions for personalized recommendation.The approach differs from existing studies by framing recommendation as instruction following by LLMs.
  • Conclusion: The method combines preference, intention, task form, and context in instruction templates, then generates 252K fine-grained personalized instructions.The instructions are used to adapt an open-source 3B Flan-T5 model to recommender systems.
  • Conclusion: Extensive experiments demonstrate effectiveness and generalization across various recommendation scenarios.The conclusion summarizes the reported empirical support for the approach.
  • Future Work: Future work includes scaling the LLM, extending context length for long behavior sequences, and supporting multi-turn user-system interaction.The authors specifically mention chit-chat-style communication as a future interaction setting.

A INSTRUCTION TEMPLATES FOR TRADITIONAL RECOMMENDATION

The appendix presents instruction templates for traditional recommendation tasks. These templates express user history, preferences, candidate items, and prediction or recommendation objectives in natural language.

  • Sequential Recommendation: One template asks the model to predict the next product from a user’s historical interactions.The task uses the interaction history as input and targets the product most likely to receive the next interaction.
  • Preference-Aware Recommendation: Other templates combine historical interactions with explicit preferences to recommend the next suitable product.These formulations ask the model to match candidate products to the user’s inferred or stated preferences.
  • Candidate Selection: Candidate-selection templates instruct the model to compare candidate items with the user’s behavioral history and preference.The output is selected from a provided candidate set.
  • Interaction Prediction: Binary templates ask whether a user is likely to interact with a specified candidate item next.The prediction is based on historical purchases or interactions and personalized preferences.
  • Preference-Only Recommendation: Preference-only templates ask the recommender to suggest products consistent with the user’s stated preferences.These templates frame recommendation as selecting or predicting products driven by explicit preference information.

B INSTRUCTION TEMPLATES FOR TRADITIONAL PRODUCT SEARCH

These templates frame traditional product search as natural-language interactions with a search engine. They cover generating related items from user queries and reversing the task to generate queries for target items.

  • Traditional search templates ask a search engine to generate an item related to a user’s query.
  • The templates support queries expressing explicit preferences, vague intentions, or specific intentions.
  • Reverse-search templates generate a query when the user wants to search for a specific target item but cannot phrase the query.
  • The reverse formulation estimates likely query keywords for a target item, including brand or type-related terms.

C INSTRUCTION TEMPLATES FOR PERSONALIZED SEARCH

These templates personalize search by combining a user’s interaction history with preferences, intentions, queries, or target items. They also support inferring preferences and selecting candidates that best match the resulting needs.

  • Personalized-search templates combine historical interactions with explicit preferences to generate products consistent with the user’s preferences.
  • Other templates use historical interactions together with vague or specific intentions to recommend products matching current needs.
  • Reverse personalized search analyzes historical interactions and a target item to identify preferences associated with that item.
  • Templates infer preference-consistent historical interactions from a user’s search query and final product choice.
  • Some templates ask the system to estimate preferences from historical purchases and use them with a search request to recommend among candidate items.
  • Candidate-selection templates present products based on inferred preferences and ask the system to choose the item best meeting the user’s needs.
Loading 2305.07001v1…