Towards Next-Generation Recommender Systems: A Benchmark for Personalized Recommendation Assistant with LLMs

1.1 Overview

• RecBench+ uses Amazon-Book and MovieLens-1M to evaluate condition-based and user profile-based recommendation queries.MovieLens-1M includes essential user demographic data.Source: §1.1 · p. 3, §1.1.1 · p. 3

1.1.1 Condition-based Query.

• Condition-based queries test constraint satisfaction through explicit conditions, implicit conditions requiring inference, and misinformed conditions containing errors.Explicit constraints are directly stated; implicit ones require multi-hop reasoning to infer the target condition.Source: §1.1.1 · p. 3

1.1.2 User Profile-based Query.

• User Profile-based queries assess personalization from interests or demographics rather than specific filtering conditions.Interests represent dynamic preferences, while demographics capture more stable attributes such as gender or occupation.Source: §1.1.2 · pp. 3–4

1.2 Condition-based Query Construction

• Knowledge graphs connect items to attributes and interdependencies, supporting realistic, diverse condition-based queries from user behavior.The construction pipeline builds an item KG, extracts shared relations, then uses an LLM to simulate users and generate queries; qualifying items provide ground truth.Source: §1.2 · p. 4

1.2.1 Item KG Construction.

• Movie KGs use Wikipedia attributes, while book KGs use Amazon Book Dataset metadata; both are combined with recommendation datasets.The movie graph covers seven attributes, including directors, actors, composers, and genres.Source: §1.2.1 · p. 4

1.2.2 Shared Relation Extraction.

• KG retrieval finds subsets of a user’s history sharing attributes, represented as relation-type and target-value pairs that become query conditions.Examples include items sharing a director or actor.Source: §1.2.2 · p. 4

1.2.3 Query Generation.

• Explicit conditions use extracted relations directly, implicit conditions replace values with references to related items, and misinformed conditions add false relations before GPT-4o generates queries.For misinformed conditions, selected KG items do not actually have the specified relation to the target value.Source: §1.2.3 · p. 5
• Each generated sample pairs a query with interaction history excluding qualifying items and the ground-truth items that satisfy its conditions.Item attributes are represented in the KG.Source: §1.2.3 · p. 5

1.3 User Profile-based Query Construction

• User Profile-based Queries generate personalized recommendations from profile information without requiring specific filtering conditions.They comprise Interest-based and Demographics-based queries.Source: §1.3 · p. 5

1.3.1 Interest-based Query.

• Interest-based queries use frequent consecutive item sequences and the sequences immediately preceding them in user histories to represent shared interests.An LLM analyzes these patterns to infer the reasoning behind common interests and generate queries reflecting item context and those interests.Source: §1.3.1 · p. 5

1.3.2 Demographics-based Query.

• Demographics-based queries group users by attribute permutations, identify their top-consumed items, and use an LLM to synthesize group-specific queries.The identified popular items serve as ground truth for evaluation.Source: §1.3.2 · pp. 5–6

• Condition-based queries are filtered out when string matching finds that the query omits provided information, such as an actor or director.Source: §1.4 · p. 6
• User Profile-based queries are manually checked, and those deemed too broad or unreasonable are removed.Source: §1.4 · p. 6

2.1.1 Baselines.

• The evaluation includes GPT-4o, GPT-4o-mini, Llama, Gemini, Claude, DeepSeek-V3, and DeepSeek-R1, plus SFT and GRPO-based RFT of two smaller models.The main testing prompts do not specify a fixed number of recommendations; the evaluation uses Precision, Recall, CMR, and FTR.Source: §2.1.1 · p. 6, §2.1.2 · p. 6

2.2 Results on Condition-based Query

• GPT-4o achieves the highest Precision and second-highest CMR average, while DeepSeek-R1 leads Recall and performs best on implicit-condition queries; explicit queries are easier and misinformed queries hardest.Misinformed-condition queries have the lowest Recall and CMR, while higher FTR may indicate that models detect incorrect information and avoid inaccurate recommendations.Source: §2.2 · pp. 6–7
• More query conditions improve Precision and Recall, but CMR falls for explicit queries and rises for implicit and misinformed queries as added context affects constraint satisfaction differently.Additional conditions narrow the candidate pool; for explicit queries, however, satisfying all constraints becomes harder.Source: §2.2 · p. 7
• User-item history improves Precision across condition-query types, but may introduce distractors that reduce strict condition adherence.A history-based genre preference can prompt a recommendation that fails the query’s stated constraint.Source: §2.2 · p. 7
• With fixed K=5, performance still declines from explicit to implicit to misinformed condition queries; two-stage SFT+RFT reaches Recall 0.410 on movie explicit queries.That Recall exceeds SFT-only at 0.391 and RFT-only at 0.178; RFT alone performs worse than SFT, while SFT warm-up followed by RFT further improves performance.Source: §2.2 · pp. 7–8, Table 3 · p. 7

2.3 Results on User Profile-based Query

• For user-profile queries, demographic-based queries generally have lower Recall than interest-based queries, and models perform better on widely shared interests except for books.For books, popular titles can have more editions or publishers, making exact matches harder; less popular titles may have fewer variants.Source: §2.3 · p. 8
• Demographic-query performance varies by user group: models show higher accuracy for female users and higher Recall for ages 50–55 than for users aged 56 and above.The paper suggests older users may be harder to model because of less online activity and less training data.Source: §2.3 · p. 8
• All fine-tuned user-profile models outperform their base models, with SFT+RFT strongest overall and SFT close behind.For Qwen-2.5-3B demographics queries, RFT achieves Recall 0.228 versus SFT’s 0.154.Source: Table 6 · p. 8, §2.3 · p. 9

3 Related Work

• Traditional benchmarks such as LLMRec [18] evaluate rating prediction and sequential recommendation, comparing LLMs with classical methods.Other recommendation benchmarks assess utility, novelty, history-length sensitivity, hallucination [13], or personalization accuracy [29].Source: §3 · p. 9
• Assistant benchmarks include general-purpose Gaia [20] and domain-specific VoiceBench [5] and Mobile-Bench [3].The paper identifies evaluation of personalized recommendation assistants as a remaining gap.Source: §3 · p. 9

4 Conclusion

• RecBench+ is a benchmark dataset for evaluating LLMs as personalized recommendation assistants.The paper frames it as a new paradigm for next-generation recommender systems.Source: §4 · p. 9
• The benchmark contains five query types designed to assess handling of complex user queries in real-world scenarios.Source: §4 · p. 9
• Experiments with diverse LLMs, including models with advanced reasoning capabilities, yield 11 observations about assistants’ strengths and limitations.Source: §4 · p. 9
• The paper also explores the impact of different fine-tuning strategies on recommendation performance.Source: §4 · p. 9

A Prompts for Model Testing

• Without interaction history, the prompt asks the model to recommend movies directly from the user's query.Source: A Prompts for Model Testing · p. 9
• With interaction history, the prompt includes watched movies or reading history and asks for recommendations satisfying both the query and user preferences.Source: A Prompts for Model Testing · p. 9
• The history-based prompt is phrased for movies or books, while the no-history prompt is shown for movies.Source: A Prompts for Model Testing · p. 9

B Case Study

• DeepSeek-R1 verifies the cinematographer, identifies Owen Roizman, and gives accurate recommendations, while DeepSeek-V3 incorrectly lists films by Scott Ambrozy.The example query asks for movies sharing the cinematographer of two given films.Source: B Case Study · p. 9

Ethical Considerations

• Human annotators checked all dataset data for harmful content; the authors state that the dataset poses no potential harm to human society.The dataset was generated using an LLM and public recommendation datasets.Source: Ethical Considerations · p. 10