PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Brand Retrieval and Ranking in Large Language Model Recommendations

Edward Malthouse, Kun-Yu Lee, Jing Yang, Sanchary Pal, Xueyan Feng

arXiv:2609.16304v1cs.IR

TL;DR

LLM recommendations are difficult to evaluate because models generate brands and rankings variably without a fixed candidate set. This paper defines competitors independently of model outputs and estimates prevalence and prominence through repeated sampling; its analyses find frequent omissions of established brands and prominence associated with marketplace visibility, while needs and positioning cues affect retrieval.

  • Problem

    LLMs can generate recommendations without a fixed candidate set, and repeated requests can return different brands and rankings, complicating evaluation.

    Source: §1 · p. 1

  • Method

    The framework defines the competitive set independently of model outputs and uses repeated recommendations to measure prevalence with BRP and prominence with MRR.

    Source: §1 · pp. 1–2

  • Results

    Category-only prompts often omitted established brands, prominence showed limited evidence of conventional popularity bias, and search interest and online brand conversation were associated with prominence.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Needs-based context can bring omitted brands into recommendations, and distinctive positioning cues can retrieve brands absent from ordinary recommendations.

    Source: §5 · p. 9

    The illustrative needs-based and diagnostic probes used handwritten prompts for two brands and require validation across a larger, randomly selected brand set.

    Source: §5.3 · p. 10

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used for product recommendation, but evaluating their recommendations presents challenges that differ from conventional information retrieval and recommender systems. LLMs can generate recommendations without an explicit candidate set, and repeated responses to the same query can produce different brands and rankings. We introduce a framework for evaluating open-ended LLM brand recommendations that defines the competitive set independently of model outputs and estimates recommendation prevalence and prominence through repeated sampling. We operationalize these constructs using Brand Recommendation Probability (BRP@$k$) and Mean Reciprocal Rank (MRR@$k$), and apply the framework to six LLMs across five product categories. Category-only queries reveal substantial omission of established brands and limited evidence that recommendation prominence follows conventional brand popularity. Instead, prominence is associated with broader marketplace-visibility signals, particularly search interest and online brand conversation. Needs-based queries show that contextualizing users' goals and constraints changes which brands are retrieved, while diagnostic positioning probes demonstrate that brands omitted from ordinary recommendations can remain conditionally retrievable when distinctive cues are supplied. These findings highlight the need to evaluate LLM recommendation as a stochastic retrieval-and-ranking process rather than from individual generated lists. We provide open-source software and data to support reproducible evaluation of LLM-generated brand recommendations.

1 INTRODUCTION

LLM brand recommendations are difficult to evaluate because models generate recommendations without a fixed candidate set and may return different brands and rankings to repeated queries. This paper introduces a framework to measure recommendation prevalence and prominence, examine associated marketplace signals, and test how consumer needs and positioning cues affect retrieval.

1 INTRODUCTION

  • A model’s competitive set is not directly observable and can vary across repeated prompts, making it difficult to assess which brands appear, how prominently, and which competitors are omitted.These omissions matter because brands left out may never enter consumers’ consideration.Source: §1 · p. 1
  • Whether LLM recommendations reflect conventional brand popularity is unclear; marketplace visibility and brand positioning may also be associated with prominence.Needs-based prompts may bring brands into recommendations when consumers state relevant goals and constraints.Source: §1 · p. 1
  • The framework defines the competitive set independently of model outputs, making recommended and omitted brands observable.It operationalizes prevalence with Brand Recommendation Probability (BRP) and prominence with Mean Reciprocal Rank (MRR).Source: §1 · pp. 1–2
  • The study examines six LLMs across five product categories and provides open-source software and data for reproducible evaluation.It also tests needs-based prompts and diagnostic positioning probes for retrieving brands omitted from ordinary recommendations.Source: §1 · p. 2

2 CONCEPTUAL BACKGROUND AND FRAMEWORK

The framework treats LLM recommendations as generated choice sets whose membership and ordering can vary, then evaluates prevalence, prominence, marketplace associations, needs-based matching, and conditional retrievability.

2.1 LLMs as Choice Architects

  • LLMs act as emergent choice architects because they construct small recommendation sets in response to prompts rather than selecting brands from a fixed inventory.Their generated alternatives must be observed in responses rather than inferred from a shelf, search ranking, or platform inventory.Source: §2.1 · p. 2

2.2 From Consideration Sets to Generated Sets

  • A generated set is a compact set of alternatives constructed in response to a prompt, and its membership and ordering may vary across repeated queries.BRP@k measures the probability a brand appears among up to k recommendations; MRR@k incorporates its rank as well as whether it appears.Source: §2.2 · p. 2
  • Category-only prompts establish a baseline for how recommendations are distributed across competing brands without specifying consumer goals, preferences, or constraints.The study asks whether LLM-generated sets favor popular brands or show different patterns of prevalence and prominence.Source: §2.2 · p. 2

2.3 Consumer Needs and Brand Positioning

  • Needs-based prompts supply consumer goals, constraints, preferences, or intended uses that may change which brands enter the generated set and how they rank.Brand positioning differentiates brands around particular needs, use cases, and value propositions.Source: §2.3 · pp. 2–3
  • Marketplace visibility indicators are interrelated, so their relationships with recommendation prominence are explored observationally rather than treated as causal.The analysis examines marketplace correlates of MRR without proposing a directional hypothesis.Source: §2.3 · p. 3

2.4 Framework for Evaluating LLM Brand Recommendations

  • The five-step framework defines competitors independently, measures category-only recommendations, explores marketplace correlates, matches needs-based prompts, and diagnoses retrieval with positioning probes.Independent competitor definition makes brands that are rarely or never recommended observable; exploratory associations should guide testing, not causal claims about reallocating marketing expenditures.Source: §2.4 · pp. 3–4
  • Diagnostic positioning probes test conditional retrievability by making distinctive brand cues explicit, helping distinguish a brand that is known but not activated from one with an incomplete or inconsistent representation.The probes can compare the LLM’s associated needs, use cases, attributes, and points of difference with the firm’s intended positioning.Source: §2.4 · p. 4

3 METHODOLOGY

The study evaluates recommendations across five product and service categories using external brand and marketplace data, six commercial LLMs, and repeated category-only and needs-based prompts.

3 METHODOLOGY

  • Five categories were selected to represent high-involvement decisions, diverse products and services, reliable external benchmarks, and sufficiently rich competitive sets.They were cordless drills, boat cruises, cat food, coffee makers, and hiking jackets, with specific consumer-facing prompts aligned to broader category data.Source: §3 · p. 4
  • The competitive set combined major brands identified by BrandZ and Statista market-share data with additional brands found in preliminary LLM recommendation trials.Marketplace indicators covered paid advertising, press mentions, online brand conversation, Google search interest, and Wikipedia page views.Source: §3 · p. 4

3.1 Experiment

  • Six commercially available LLMs were accessed through provider APIs using stateless requests with no conversation history, user information, or external retrieval tools.The six models were GPT-5.5, GPT-5.4 Mini, Gemini 3.1 Pro Preview, Gemini 2.5 Flash, Claude Opus 4.7, and Claude Sonnet 4.6.Source: §3.1 · p. 5
  • Category-only prompts were issued 40 times per category and model, while 100 needs-based prompts were each submitted to all six models twice.The two stages each produced 1,200 recommendation lists; responses were standardized to the first five unique brands using category-specific alias dictionaries.Source: §3.1 · p. 5

4 RESULTS

Category-only recommendations omit established brands and show limited alignment with conventional brand popularity, while prominence is associated with marketplace-visibility signals. Needs-based prompts can retrieve some previously omitted brands, and diagnostic cues can make both focal brands conditionally retrievable.

4.1 Measuring Brand Recommendation Prevalence and Prominence

  • Many established brands receive no category-only recommendations, making omission—not just low rank—a material result.Examples span hiking jackets, drills, coffee makers, cat food, and cruises.Source: §4.1 · p. 5
  • BrandZ salience shows little systematic relationship with prominence; premium brands lead in some categories, but that pattern is not universal.Even in cruises, where two highly salient brands rank prominently, Viking is prominent despite lower salience and Disney Cruise Line is almost never recommended.Source: §4.1 · p. 5
  • BRP@5 indicates whether brands consistently enter recommendation sets, while MRR distinguishes their prominence when prevalence is similar.For cordless drills, five brands have BRP@5=1, yet DeWalt ranks highest and Bosch and Ryobi lower.Source: §4.1 · pp. 5–6

4.2 Explaining Brand Recommendation Prominence

  • The prominence analysis uses a fractional-logit model with ridge regularization because visibility predictors are correlated and MRR is bounded with many zero values.The authors standardize five marketplace measures and include category controls; they treat the analysis as exploratory observational work, not a confirmatory or causal model.Source: §4.2 · p. 7
  • At the cross-validated ridge setting, search interest has the largest coefficient for MRR, followed by online brand conversation; the ordering is robust across specifications.News, Wikipedia page views, and advertising expenditure have coefficients close to zero; the authors characterize the relationships as exploratory and predictive, not causal.Source: Figure 5 · p. 8, §4.2 · p. 8

4.3 Needs-based Prompts

  • Needs-based prompts retrieved Craftsman in 35.4% of lists and L.L.Bean in 5.4%, after neither appeared in category-only recommendations.Each estimate is based on 240 lists, and the authors describe the examples as illustrative rather than a systematic comparison across brands.Source: §4.3 · pp. 8–9
  • Diagnostic positioning cues raised BRP@5 to 81.3% for Craftsman and 88.5% for L.L.Bean, showing conditional retrievability but not complete or accurate brand representations.The probes used distinctive language from the brands’ marketing materials and were intentionally unlike typical consumer requests.Source: §4.3 · p. 9

5 DISCUSSION

Across categories, LLMs often omit established brands, and recommendation prominence shows limited correspondence with conventional brand popularity. Needs-based prompts and diagnostic positioning cues can change retrieval, while marketplace visibility is associated with prominence.

5 DISCUSSION

  • Category-only recommendations often omit established brands, show limited evidence of conventional popularity bias, and vary in prominence across categories.In cordless drills and hiking jackets, recommendations tend to favor premium brands over mass-market alternatives.Source: §5 · p. 9
  • Needs-based prompts can bring omitted brands into recommendation sets, while diagnostic positioning cues make Craftsman and L.L.Bean highly retrievable.These illustrative results suggest that low recommendation rates do not necessarily mean brand knowledge is absent.Source: §5 · p. 9, §5.1 · pp. 9–10

5.1 Theoretical Contributions

  • The generated set is a variable, LLM-constructed alternative set, so Brand Recommendation Probability measures inclusion and Mean Reciprocal Rank measures prominence across repeated queries.Both measures require sampling because membership and ordering can vary across responses to the same prompt.Source: §5.1 · p. 9
  • BrandZ salience has limited, category-varying correspondence with recommendation prominence, which is associated instead with broader marketplace visibility, especially search interest and online conversation.The observed relationships are correlated and observational, so their mechanisms cannot be identified.Source: §5.1 · p. 9

5.2 Practical Implications

  • For managers, recommendation prominence is associated most strongly with Google search interest and then online conversation, but these signals are not causal optimization targets.The authors suggest investigating activities that generate sustained marketplace attention.Source: §5.2 · p. 10
  • Managers should communicate category membership and distinctive points of difference consistently, linking brand associations to the consumer needs the brand is intended to serve.The paper argues that coherent positioning across controlled and influenceable touchpoints may make brands easier for LLMs to match to articulated needs.Source: §5.2 · p. 10

5.3 Limitations and Future Research

  • Illustrative positioning probes cover two brands with handwritten prompts and need broader validation; fresh sessions also omit consumers’ prior chat histories.The price-tier pattern appeared in only two of five categories, and the study does not establish what distinguishes those categories.Source: §5.3 · p. 10

A ADDITIONAL DATA DETAILS

The additional-data material identifies tables for data sources and category disambiguation, alongside a note on exact-phrase matching. The supplied reference-list excerpts include work on consumer choice, recommender systems, and econometric methods.

A ADDITIONAL DATA DETAILS

Loading 2609.16304v1…