PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Utility: Evaluating LLM as Recommender

Chumeng Jiang, Jiayin Wang, Weizhi Ma, Charles L. A. Clarke, Shuai Wang, Chuhan Wu, Min Zhang

arXiv:2411.00331v2cs.IR

TL;DR

Existing evaluations of LLM recommenders give limited attention to LLM-specific evaluation aspects. The paper introduces a multidimensional framework and evaluates LLM and traditional recommenders across ranking and re-ranking; LLMs perform strongly in familiar ranking domains and better in re-ranking, while exhibiting substantial position bias.

  • Problem

    Existing evaluations often omit LLM-specific aspects, limiting assessment of characteristics such as position bias, hallucinations, and generated user profiles.

    Source: §2.2 · p. 2

  • Method

    The paper proposes a framework combining utility and novelty with four LLM-related dimensions and evaluates LLM recommenders in ranking and re-ranking settings.

    Source: §1 · p. 2

  • Results

    LLMs excel in ranking domains where they have more knowledge, while in re-ranking they outperform traditional recommenders in nearly all conventional dimensions; candidate position bias remains substantial.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework and observations are intended to support a more comprehensive understanding of LLM recommendations and their differences from traditional models.

    Source: §1 · p. 2

    The framework requires recommendation data that includes item titles and user-item interactions.

    Source: §3.3 · p. 3

Abstract

from arXiv · show

With the rapid development of Large Language Models (LLMs), recent studies employed LLMs as recommenders to provide personalized information services for distinct users. Despite efforts to improve the accuracy of LLM-based recommendation models, relatively little attention is paid to beyond-utility dimensions. Moreover, there are unique evaluation aspects of LLM-based recommendation models, which have been largely ignored. To bridge this gap, we explore four new evaluation dimensions and propose a multidimensional evaluation framework. The new evaluation dimensions include: 1) history length sensitivity, 2) candidate position bias, 3) generation-involved performance, and 4) hallucinations. All four dimensions have the potential to impact performance, but are largely unnecessary for consideration in traditional systems. Using this multidimensional evaluation framework, along with traditional aspects, we evaluate the performance of seven LLM-based recommenders, with three prompting strategies, comparing them with six traditional models on both ranking and re-ranking tasks on four datasets. We find that LLMs excel at handling tasks with prior knowledge and shorter input histories in the ranking setting, and perform better in the re-ranking setting, beating traditional models across multiple dimensions. However, LLMs exhibit substantial candidate position bias issues, and some models hallucinate nonexistent items much more often than others. We intend our evaluation framework and observations to benefit future research on the use of LLMs as recommenders. The code and data are available at https://github.com/JiangDeccc/EvaLLMasRecommender.

1 Introduction

Prior evaluations emphasize conventional dimensions, but LLM recommenders also raise questions tied to their distinctive capabilities and failure modes. The paper proposes a multidimensional framework and reports contrasting results across ranking and re-ranking settings.

1 Introduction

  • Conventional evaluations cover accuracy and other dimensions, but prior work assesses them under different settings without one comprehensive assessment.LLM-specific concerns include position bias and hallucinations, alongside opportunities from textual and generative capabilities.Source: §1 · pp. 1–2
  • The framework adds history length sensitivity, candidate position bias, generation-involved performance, and hallucinations to utility and novelty.The first two dimensions can also be assessed for traditional models, while generation-involved performance and hallucinations are specific to LLM recommenders.Source: §1 · p. 2
  • LLMs can excel in familiar domains and cold-start ranking, but substantial candidate position bias damages recommendation quality; re-ranking performance exceeds traditional recommenders in nearly all conventional dimensions.The authors also report that position bias is partially alleviated in re-ranking compared with ranking.Source: §1 · p. 2
  • The evaluation covers seven LLMs, three prompting strategies, six traditional models, and four datasets across ranking and re-ranking tasks.The authors describe the framework as reproducible and report seven observations across the four LLM-related dimensions.Source: §1 · p. 2

2 Related Work

Prior LLM-as-recommender work develops prompt strategies and recommendation-data tuning, while evaluation studies mostly emphasize conventional dimensions. The paper motivates examining LLM-specific characteristics alongside those established measures.

2.1 LLM as Recommender

  • Prior LLM-as-recommender work explores manual and automated prompting, in-context learning, and tuning on recommendation data to align language and recommendation or item spaces.OpenP5 and POD unify recommendation tasks with templates; other studies explore item indexing to incorporate collaborative information.Source: §2.1 · p. 2

2.2 Evaluation of Recommender Systems

  • Existing evaluations often focus on conventional dimensions, including novelty, popularity bias, diversity, and fairness, across varied tasks and user or item fairness settings.The cited work includes ChatGPT top-K, cold-start, and re-ranking evaluations, as well as user-group and item-side fairness studies.Source: §2.2 · p. 2
  • Conventional evaluation alone misses LLM-specific concerns: candidate-position preference, nonexistent-item hallucinations, and generated user profiles that may affect recommendations.The authors argue these characteristics require closer examination for LLM recommenders and are not concerns for traditional recommendation models.Source: §2.2 · p. 2

3 Multidimensional Evaluation

The framework evaluates LLM recommenders across ranking and re-ranking tasks using conventional and LLM-related dimensions. It specifies candidate-set construction, input-data handling, and measures for recommendation quality and LLM-specific effects.

3.1 Framework Overview

  • The framework combines utility and novelty with four LLM-related dimensions, evaluating how LLM characteristics may affect recommendation performance.The four dimensions are history length sensitivity, candidate position bias, generation-involved performance, and hallucinations.Source: §3.1 · p. 3, §3.4 · p. 4

3.2 Task Settings

  • Ranking forms candidates from a positive item and randomly sampled negatives, while re-ranking orders items aggregated from traditional models’ top-K recommendations.Re-ranking refines preliminary recommendations; its candidate set is aggregated from recall models trained on the training data.Source: §3.2 · p. 3

3.3 Input Data

  • The framework accepts datasets with item titles and user-item interactions, and supports small test samples selected with a K-S test for distributional similarity.It uses leave-one-out splitting and resamples users until the K-S test finds no significant difference from the full dataset.Source: §3.3 · pp. 3–4

3.4 Evaluation Dimensions

  • Utility measures recommendation accuracy with HR and NDCG, while novelty assesses recommendations of less-popular items and unexpected, useful items.APLT measures exposure of long-tail items, defined as the bottom 80% by item frequency; serendipity counts correct recommendations absent from MostPop’s recommendations.Source: §3.4 · p. 4
  • History length sensitivity compares accuracy across history lengths, while candidate position bias compares random positive-item placement with placement first.The position-bias metric applies an inverse logarithmic transformation to accuracy before comparing variation; ideally, it is zero.Source: §3.4 · p. 4
  • Optional LLM-generated profiles can be added to prompts or replace the original history; hallucination measurement counts recommended titles unmatched to catalog items.Profiles are generated from a specified length of interaction history, and the framework compares recommendation performance with and without profiling.Source: §3.4 · pp. 4–5

4 Evaluation Results

Across ranking and re-ranking, LLM recommenders show strengths in knowledge-rich settings and niche-item recommendation, but also display history-length sensitivity, candidate-position bias, and hallucinations. Re-ranking results are stronger than ranking results, while generated profiles capture useful user-history patterns.

4.1 Evaluation setup

  • Experiments use four real-world datasets, seven LLMs with three prompting strategies, and six traditional models; each ranking and re-ranking test samples 1,000 users with 20 candidates.The datasets are Amazon All Beauty, Amazon Sports & Outdoors, Movielens-1M, and LastFM; re-ranking candidates come from four traditional recall models.Source: §4.1.1 · p. 5, §4.1.2 · p. 5, §4.1.3 · p. 5

4.2 Utility

  • LLMs generally trail traditional models in ranking accuracy, but can exceed them in domains where they have extensive knowledge, as in LastFM.In Sports, GPT-4o outperforms GRU4Rec but trails LightGCN; in LastFM, several LLMs beat LightGCN.Source: §4.2 · p. 5

4.3 Novelty

  • LLMs are less affected by popularity bias than traditional models, and leading LLMs can outperform traditional models on serendipity in Beauty.In LastFM, GPT4o combines better accuracy with less popularity bias; the authors characterize LLMs as adept at correctly recommending niche items.Source: §4.3 · pp. 5–6, Figure 2 · p. 5

4.4 History Length Sensitivity

  • LLMs often peak with fewer than 10 history inputs, while traditional models improve with longer histories; some LLMs also beat traditional models with only 1–2 records.Without history, LLM HR@5 is near random (0.25) in Beauty and Sports, but exceeds 0.25 in ML-1M and LastFM; Qwen2 and Mistral deteriorate with history in ML-1M.Source: §4.4 · p. 6, Figure 3 · p. 6

4.5 Candidate Position Bias

  • LLM recommendation accuracy generally declines as positive items appear later in candidate lists, whereas traditional models remain relatively steady overall.Mistral is relatively stable across the first three datasets but favors later positions in LastFM; the authors warn that preference for early items can reduce positive-item priority.Source: §4.5 · pp. 6–7, Figure 4 · p. 7

4.6 Generation-Involved Performance

  • In Beauty and Sports, generated profiles nearly preserve history-only accuracy, and using profile plus history improves all four dimensions; in ML-1M and LastFM, profiles can reduce accuracy.The authors report that generated profiles capture key user-history patterns and may enhance interpretability.Source: §4.6 · p. 7, Figure 5 · p. 7

4.7 Hallucinations

  • Most LLMs generate fewer than 5% nonexistent items, but Mistral produces about 23% in Beauty, with hallucinations threatening recommendation quality and user experience.Errors include omitting title parts, combining titles, or misstating item details, making some generated items difficult to map to dataset items.Source: §4.7 · p. 7, Table 3 · p. 7

4.8 LLM as Re-Ranker

  • GPT-3.5 outperforms four traditional models in re-ranking utility and niche-item exposure, while LLM candidate-position bias is relatively alleviated.The authors suggest LLMs may use features beyond those available to the recall models; ordering candidates by the original recall ranking may leverage position bias.Source: §4.8 · pp. 7–8, Table 4 · p. 8

5 Discussions

Item grounding’s benefit depends on hallucination rate, while fine-tuning reduces hallucinations more effectively than candidate position bias. Both issues can persist after fine-tuning.

5.1 Impact of Item Grounding

  • Below a hallucination rate of 0.005, almost all grounding strategies fail to improve performance; above 0.005, BM25 outperforms LLaMA in most cases.The reported grounding comparison is on ML1M.Source: §5.1 · p. 8, Table 5 · p. 8
  • For low hallucination rates, random grounding may suffice; at higher rates, the choice between the best embedding model and retrieval-based grounding depends on cost priorities.The authors recommend the best embedding model when cost is not primary and retrieval-based strategies when cost efficiency matters.Source: §5.1 · p. 8

5.2 Impact of LLM Finetuning

  • Fine-tuning reduced ML1M hallucination from 0.425 to 0.001, but its mitigation of candidate position bias was less effective and the bias could persist.The study fine-tuned LLaMA3.1-8B-Intruct for up to three epochs across four datasets; candidate position bias improved on Beauty, Sports, and LastFM but worsened slightly on ML1M.Source: §5.2 · p. 8, Table 6 · p. 8

6 Conclusions

The conclusion argues that LLM recommenders need evaluation beyond conventional utility and novelty. Across ranking and re-ranking, the results show both distinctive strengths and persistent challenges.

6 Conclusions

  • LLMs perform well in familiar domains and with relatively short histories in ranking, show severe candidate position bias, and perform better across multiple dimensions in re-ranking.The authors point to longer histories and item-mapping strategies as future directions.Source: §6 · p. 8

A Example of Generated User Profile

The examples illustrate both how generated profiles summarize user preferences and how LLM recommenders can use item text effectively yet be drawn to clickbait.

A Example of Generated User Profile

B Case Study

  • In one case, title similarity helps LLMs select the right low-popularity item where traditional models perform worse; in another, long, complex clickbait titles receive top priority.The clickbait example contains multiple appealing elements, including natural ingredients, large capacity, and multifunctionality.Source: B Case Study · p. 10

C.1 Dataset

The appendix describes four real-world datasets, their preprocessing and split, and the traditional models used for comparison.

C.1 Dataset

  • The study uses Amazon All Beauty, Amazon Sports & Outdoors, MovieLens-1M, and LastFM, with item titles as descriptions, 5-core filtering, and leave-one-out splitting.Dataset statistics are presented in Table 7.Source: §C.1 · pp. 10–11, Table 7 · p. 11

C.2 A Brief Introduction of Traditional Models

  • The traditional comparison models include MostPop, BM25, BPRMF, GRU4Rec, SASRec, and LightGCN, representing popularity, ranking, pairwise, sequential, and graph-based approaches.The passage describes BM25 as TF-IDF-based, BPRMF as optimizing a pairwise ranking loss, GRU4Rec as RNN-based, SASRec as self-attention-based, and LightGCN as graph-based.Source: §C.2 · p. 11

C.3 Implementation Details

The implementation adapts sequential-recommendation prompts and evaluates ranking and re-ranking with controlled candidate pools, repeated runs, and model-specific hyperparameter searches. The appendix also lists the reported settings for the traditional models.

C.3 Implementation Details

  • LLM prompt templates adapt OpenP5 sequential-recommendation templates with added output formatting, while traditional models use ReChorus and detailed model-specific hyperparameter searches.Reported results average five runs with different random initializations; tests randomly sample 1,000 users, and LLM temperatures are set to the lowest valid values: 0 for GPT and LlaMa, 0.01 for GLM.Source: §C.3 · p. 11
  • Ranking tests use one positive and 19 randomly retrieved negatives per user, with candidate positions shuffled but held consistent across LLM experiments.Source: §C.3 · p. 11
  • Re-ranking uses 20-item pools drawn from the top-K recommendations of BPRMF, GRU4Rec, SASRec, and LightGCN, keeping each pool's order unchanged.Source: §C.3 · p. 11

C.4 Hyperparameter Settings

  • The reported traditional-model configurations follow detailed hyperparameter searches, with settings enumerated for BM25, BPRMF, GRU4Rec, SASRec, and LightGCN.The listed settings vary by dataset and model.Source: §C.4 · pp. 11–12

D Metric Definition

This appendix surveys the evaluation metrics used in the framework, covering diversity, fairness, novelty, and popularity, with notation summarized in Table 8.

  • The framework's metric appendix provides an overview of evaluation measures and points readers to Table 8 for frequently used notation.Source: D Metric Definition · p. 12, Table 8 · p. 12
  • Self-information reflects how likely recommended items are already known to users, while Serendipity compares recommendations with Most Popular and incorporates unexpectedness and usefulness.Higher values on both metrics indicate better unpredictability.Source: D Metric Definition · p. 12
  • ARP measures average popularity in top-K lists, while APLT measures the share of long-tail items, defined as the bottom 80% by popularity.The paper motivates exposure of niche items because leading popular items can dominate recommendations and leave users with niche hobbies unsatisfied.Source: D Metric Definition · p. 12
  • PopREO measures differences in true-positive rates across five item-popularity groups; lower values indicate less popularity bias in recommendations.Source: D Metric Definition · p. 12
  • Item Coverage measures the share of candidate items appearing in recommendation lists, while OIC measures overlap across users' lists and higher OIC indicates poorer personalization and inter-list diversity.Source: D Metric Definition · pp. 12–13
  • Fairness measures cover item-side Gini, active-versus-inactive user Demographic Parity Difference, and user-level Jain's Index; group-level item fairness is omitted because PopREO addresses popularity groups.Active and inactive users are divided by median interaction-history length; smaller Gini and DPD indicate fairer outcomes, while Jain's Index is better closer to 1.Source: D Metric Definition · pp. 12–13

Related papers

Loading 2411.00331v2…