PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders

Yupeng Hou, Jiacheng Li, Xiangjun Fu, Zhankui He, An Yan, Xiusi Chen, Julian McAuley

arXiv:2403.03952v2cs.IR

TL;DR

Textual item features are difficult to use effectively in recommendation, and general embedding benchmarks may not reflect recommendation-specific encoder requirements. BLAIR introduces a large-scale dataset and unified benchmark covering multiple recommendation scenarios, then finds little correlation between LLM rankings on BLAIR and MTEB.

  • Problem

    Recommendation research lacks clear understanding of LLMs as semantic encoders, while model selection often relies on general embedding benchmarks whose tasks differ from recommendation.

  • Method

    BLAIR evaluates LLM-generated semantic representations across sequential recommendation, collaborative filtering, and product search using Amazon Reviews 2023 and complex-query datasets.

  • Results

    LLM rankings on BLAIR show little correlation with MTEB rankings, while semi-synthetic Amazon-C4 and real-world Reddit-Movie scores correlate at 0.94 for complex-query search.

  • Takeaways & Limitations

    Recommendation-specific evaluation is needed because semantic encoders face challenges distinct from those measured by general text embedding benchmarks.

  • Takeaways & Limitations

    The benchmark is limited to English data, 11 LLMs, and a subset of Amazon Reviews 2023 categories.

Abstract

from arXiv · show

Feature engineering has long been central to recommender systems, yet effectively leveraging textual item features remains challenging. Recent advances in large language models (LLMs) have enabled their use as semantic encoders for recommendation, but their roles and behaviors in this setting are still not well understood. Prior studies often rely on general-purpose embedding benchmarks (e.g., MTEB) when selecting LLMs, overlooking the unique characteristics of recommendation tasks. To address this gap, we introduce BLaIR, a comprehensive benchmark for evaluating LLMs as semantic encoders in recommendation scenarios. We contribute (1) a new large-scale Amazon Reviews 2023 dataset with over 570 million reviews and 48 million items, (2) a unified benchmark covering sequential recommendation, collaborative filtering, and product search, and (3) a new complex-query product search task featuring both semi-synthetic and real-world evaluation datasets. Experiments with 11 leading LLMs show that their rankings on BLaIR show little correlation with MTEB, highlighting the unique challenges of semantic encoding in recommendation.

1 Introduction

Textual item features are semantically rich but difficult to integrate into recommendation models, and LLM-based encoders remain insufficiently understood in this setting. BLAIR addresses this gap with a benchmark spanning recommendation scenarios and finds weak alignment between recommendation and general embedding evaluations.

  • Motivation: Textual item features are noisy, unstructured, and difficult to integrate directly, while earlier keyword and shallow statistical methods discarded semantic information.These approaches also lacked the broader common-sense knowledge inherent in natural language.
  • Motivation: LLMs are increasingly used to encode text representations as inputs for downstream recommendation models, but their role as semantic encoders remains not well understood.Recommendation representations are consumed by downstream models, unlike many generic embedding applications.
  • BLAIR benchmark: BLAIR evaluates LLM semantic encoders across sequential recommendation, collaborative filtering, and product search with short and complex queries.The benchmark converts item text features into semantic representations used to train downstream recommendation models.
  • Contributions: BLAIR introduces Amazon Reviews 2023, containing over 570 million reviews and 48 million items, alongside a unified recommendation benchmark and complex-query search task.The complex-query task includes semi-synthetic and real-world evaluation datasets.
  • Findings: LLM rankings on BLAIR are not positively correlated with MTEB rankings, showing that recommendation poses distinct semantic-encoding challenges.The benchmark also reports that semantic-encoder performance scales with encoder size, with weaker effects as task complexity grows, and that semi-synthetic evaluation correlates strongly with real-world evaluation.

2 Related Work

Prior work progressed from keyword-based and shallow content modeling toward pretrained language encoders, while separate benchmarks evaluated embeddings across general tasks. BLAIR differs by directly benchmarking LLMs as textual-feature encoders for recommendation scenarios.

  • Semantic encoders in recommendation: Early content-based recommendation methods extracted keywords and learned collaborative relations between users and items.
  • Semantic encoders in recommendation: Later studies encoded text features with pretrained models to capture language semantics, but domain gaps remained between pretraining data and recommendation scenarios.
  • Benchmarking text embeddings: BEIR, MTEB, and MMTEB aggregate datasets to evaluate text embeddings across retrieval and broader applications such as classification and clustering.
  • Benchmarking LLMs for recommendation: Existing LLM recommendation benchmarks study web interaction, shopping knowledge, user intent, or LLMs as recommendation models.
  • Benchmarking LLMs for recommendation: BLAIR is presented as the first benchmark, to the authors’ knowledge, for LLMs encoding textual features in recommendation scenarios.

3 A Large-Scale E-Commerce Dataset: Amazon Reviews 2023

Existing Amazon Reviews datasets are widely used but outdated, motivating Amazon Reviews 2023 as a larger, newer, and richer resource for recommendation benchmarking.

  • Motivation: Amazon Reviews datasets provide rich item features and are representative public resources for evaluating recommendation models.
  • Motivation: The most recent earlier Amazon Reviews dataset was last updated in 2018, limiting its recency for current benchmarking.
  • Dataset improvements: Amazon Reviews 2023 contains 3.18× more items and 2.58× more review and metadata tokens than the 2018 version.
  • Dataset improvements: The new dataset adds reviews from October 2018 to September 2023 and provides re-parsed structured JSON metadata with richer descriptive and multimodal fields.
  • Dataset improvements: Amazon Reviews 2023 also provides finer-grained timestamps, addressing inaccuracies relevant to time-sensitive tasks such as sequential recommendation.

4 BLAIR Benchmark

BLAIR benchmarks LLMs as semantic encoders across sequential recommendation, collaborative filtering, and product search. It also introduces complex-query search with semi-synthetic and real-world datasets.

  • Benchmark scope: BLAIR covers sequential recommendation, collaborative filtering, and product search, with complex-query product search as a newly introduced subtask.The complex-query setting uses longer, more descriptive, and often ambiguous queries.
  • Semantic encoding pipeline: LLMs transform item text into dense representations that can feed downstream recommendation models or support direct retrieval.For sequential recommendation and collaborative filtering, adapter or projection layers align embeddings with the recommendation space; product search uses raw representations directly.
  • Sequential recommendation: Sequential recommendation predicts the next item from a time-ordered interaction sequence using projected item embeddings and a Transformer decoder.The model is trained by maximizing next-item likelihood with cross-entropy loss.
  • Collaborative filtering: Collaborative filtering predicts likely future user-item interactions from historical interactions using projected item representations and an averaged user representation.The user representation is computed from projected representations of interacted items, and training uses InfoNCE with in-batch negatives.
  • Product search: Product search retrieves relevant items by encoding both queries and items with the same LLM and ranking candidates by their representation dot product.The benchmark distinguishes conventional short queries from newly introduced complex queries and includes ESCI for the short-query setting.
  • Complex-query product search: Complex-query evaluation combines a semi-synthetic Amazon-C4 dataset with a real-world Reddit-Movie dataset because public paired query-item data are scarce.Amazon-C4 rephrases detailed Amazon reviews into first-person, query-like requests while filtering cases that reveal the target product.

5 Experiments

Experiments evaluate 11 LLM-based semantic encoders across recommendation tasks and examine benchmark alignment, scaling, metadata, and adaptor choices. Results show that recommendation-specific performance differs from general embedding performance, while task complexity and input design affect observed gains.

  • Experimental setup: 11 LLMs selected from MTEB are evaluated as semantic encoders across BLaIR recommendation tasks.The evaluated models include open-source models below and above 1B parameters and proprietary text embedding models.
  • Main results: −0.476 Spearman correlation (p = 0.233) between BLaIR and MTEB rankings shows little alignment between recommendation and general embedding performance.The authors attribute this mismatch to differences in task objectives, discriminative representation requirements, and possible MTEB-specific overfitting.
  • Main results: Larger semantic encoders generally perform better with simple collaborative-filtering architectures, but scaling becomes less pronounced with complex sequential-recommendation Transformers.Qwen3-Embedding models of different sizes show comparable sequential-recommendation performance despite controlled downstream-model parameters.
  • Main results: 0.94 Pearson correlation (p < 0.01) between Amazon-C4 and Reddit-Movie NDCG@100 scores supports Amazon-C4 as a proxy for complex-query product-search evaluation.The correlation holds across evaluated models despite the semi-synthetic dataset’s different domain coverage.
  • Performance w.r.t. Item Metadata: Adding item descriptions to titles does not consistently improve performance across the evaluated models and datasets.Longer inputs may add noise, while LLM world knowledge may already capture information present in descriptions.
  • Performance w.r.t. Adaptor Design: PCA outperforms MRL for complex downstream tasks, whereas MRL tends to perform better for simpler collaborative-filtering architectures.The authors associate PCA’s advantage with whitening and MRL’s advantage with preserving task-relevant information in low-dimensional spaces.

6 Conclusion

BLAIR benchmarks LLM semantic encoders using a new Amazon Reviews 2023 dataset, unified recommendation tasks, and complex-query product search. Across 11 LLMs, recommendation-specific rankings differ from MTEB, scaling weakens with task complexity, and Amazon-C4 correlates strongly with real-world evaluation.

  • Contributions: BLAIR introduces Amazon Reviews 2023, a unified benchmark, and a complex-query product-search task with semi-synthetic and real-world datasets.The benchmark spans sequential recommendation, collaborative filtering, and product search.
  • Key insights: LLM rankings on BLAIR show little correlation with general text-embedding benchmarks such as MTEB.This indicates that semantic encoding for recommendation poses distinct challenges.
  • Key insights: Scaling laws apply to semantic encoders under parameter-controlled downstream models, but their effect weakens as task complexity increases.
  • Key insights: Amazon-C4 strongly correlates with real-world complex-query evaluation and can serve as a reliable evaluation proxy.The conclusion also notes potential benefits for training LLMs for complex-query tasks.

Limitations

BLAIR’s evaluation scope is limited by language, model count, computational budget, and dataset coverage. Broader multilingual and category-level evaluation would be valuable for assessing generalizability.

  • Scope limitations: The benchmark focuses exclusively on English-language data, leaving multilingual evaluation for future work.The limitation is relevant to assessing cross-lingual semantic-encoder capabilities in global e-commerce and recommender systems.
  • Scope limitations: Computational budget limits experiments to 11 LLMs and a subset of Amazon Reviews 2023 categories.Evaluating more state-of-the-art models and broader categories could provide deeper generalizability insights.

Ethical Considerations

The work emphasizes privacy and data ethics by using public information, excluding user metadata, and retaining established dataset policies. It also frames BLAIR as an evaluation benchmark rather than a deployment-ready recommender system.

  • Privacy: Amazon Reviews 2023 uses publicly available information that users explicitly chose to share and excludes private or restricted content.
  • Privacy: The dataset omits user metadata to reduce risks of user profiling and privacy violations.It focuses on publicly visible product reviews and item metadata.
  • Data ethics: BLAIR tasks use publicly released datasets while following their original data-collection and usage policies.
  • Deployment scope: BLAIR evaluates semantic encoding capabilities rather than deployment-ready recommender systems, while real-world applications should consider fairness, transparency, and societal impacts.

Appendices

The authors provide a benchmarking toolkit to support reproducibility and integration of new models and datasets.

  • The toolkit implements all datasets, tasks, and evaluation metrics used in the study.It also includes scripts for model training and evaluation.
  • The toolkit is intended to simplify adding new models and datasets.The authors invite community contributions to extend the benchmark.

B Dataset Collection Details

Amazon Reviews 2023 is collected with a user-centric pipeline that aims to preserve complete individual review histories, but item-level coverage and recency are limited by the user pool and collection timing.

  • The collection pipeline samples user identifiers and iteratively gathers all review pages associated with each user.This user-level design targets completeness of individual review histories for recommendation and user modeling.
  • Review coverage for an item may be incomplete when some users are absent from the sampled user pool.This is an item-level coverage limitation of the user-centric collection design.
  • Recently posted reviews may be missing before the dataset cutoff because collection spans time.

C.1 Data Split and Preprocessing

The benchmark standardizes preprocessing and evaluation across sequential recommendation, collaborative filtering, and product search, while analyzing why recommendation-oriented rankings differ from general embedding benchmarks and where complex search remains difficult.

  • C.1 Data Split and Preprocessing: Sequential recommendation sorts interactions by timestamp and truncates histories to the most recent 50 items, or 200 for ML-1M.Amazon Reviews 2023 uses time-based 8:1:1 splits; ML-1M and Yelp use iterative filtering and leave-last-out evaluation.
  • C.1 Data Split and Preprocessing: Collaborative filtering discards timestamps and uses 4:3:3 interaction splits, with minimum-interaction filtering except for Amazon Reviews 2023 categories.The thresholds are 20 interactions for ML-1M and Book-Crossing and 5 for Yelp.
  • C.1 Data Split and Preprocessing: Product search evaluates ranked candidates using ESCI short queries and complex-query datasets with 50 in-domain sampled items per ground-truth pair.Complex-query search ranks over both in-domain and cross-domain candidates, whereas ESCI ranks over multi-domain candidates.
  • C.1 Data Split and Preprocessing: Titles are used as Amazon item metadata because adding descriptions did not consistently improve performance.Other datasets use titles with release years, business metadata fields, or book titles, depending on the dataset.
  • C.1 Data Split and Preprocessing: NDCG is the primary metric, measured at 10 for sequential recommendation, 20 for collaborative filtering, and 100 for product search.Overall comparisons use Borda Count alongside Avg. Overall and Avg. per Task; trained tasks select checkpoints by validation performance, while product search is zero-shot.
  • E Discussion: BLaIR’s weak correlation with MTEB is hypothesized to reflect distinct needs for semantic similarity, informative representations, and instruction-following with reasoning.These capabilities are associated respectively with collaborative filtering or short-query search, sequential recommendation, and complex-query product search.
  • E Discussion: Complex downstream architectures weaken encoder-scaling gains, and a Reddit-Movies failure case reports GritLM-7B at NDCG@100 of 0.0734 over 50k movies.The authors suggest that stronger intent modeling, explicit reasoning, or additional ranking signals may be needed.
Loading 2403.03952v2…