PaperlayerView exact revision on arXiv

Source-linked AI summary

Statistical Detection of Positional Bias in LLM-Based Recommender Systems: Methodologies and Empirical Evidence

Carlos Peláez-González, Francisco Herrera, Andrés Herrera-Poyatos

arXiv:2610.37962v1cs.IRcs.AI

TL;DR

In top-1, single-turn LLM recommenders, candidate order may influence recommendations independently of relevance, a behavior requiring formal measurement. The paper proposes complementary statistical tests and finds positional bias across all evaluated models. Its conclusions are scoped to the evaluated product-recommendation setting.

  • Problem

    Position may influence LLM recommendations independently of item relevance, while its manifestation and measurement in top-1 LLM recommenders remain unstudied.

    Source: §1 · p. 2

  • Method

    The paper pairs contingency-matrix independence testing (POB-ICM) with a sample-efficient uniformity-based test (POB-UNBP) to detect positional bias under permutations.

    Source: §1 · p. 3, §4.4.2 · p. 16

  • Results

    All evaluated models reject the null hypotheses of positional independence and uniformity in permutation-based evaluations.

    Source: §5 · p. 21

  • Takeaways & Limitations

    Positional bias is a measurable property of top-1, single-turn LLM recommendation in the evaluated setting, and the paper provides metrics for quantifying it.

    Source: §1 · p. 3, §5 · p. 21

    The evaluation uses template-generated queries and one fixed prompt in single-turn product recommendation, so other domains, query formulations, or prompts may show different bias patterns.

    Source: §5 · p. 21

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly used as recommendation generators in conversational recommender systems (RS) via retrieval-augmented generation (RAG). This flexible design introduces new sources of structural bias. This paper studies positional bias in top-1 and single-turn LLM-based RS: systematic dependence of the recommended item on the order in which candidates are presented in the prompt. We propose two complementary detection methodologies: (i) POB-ICM, based on contingency matrix analysis and Chi-square independence testing, and (ii) POB-UNBP, a sample-efficient approach derived from a uniformity null model under permutation-based evaluation, enabling stochastic decoding with formal statistical guarantees. Using a controlled RAG testbed for product recommendation, built on a real product catalog, a fixed prompt template and 2000 template-generated user queries with 20 permutations each (40,000 instances), both methodologies reject the null hypothesis of positional independence and uniformity in all evaluated state-of-the-art models. We introduce three complementary metrics -- permutation sensitivity, positional bias, and strong positional bias -- to quantify this effect, the latter targeting cases where the position displaces the recommendation towards clearly less relevant items rather than only choosing among comparably relevant ones. A robustness analysis across candidate-set size and decoding temperature shows that the bias is structural and model-dependent, not an artifact of deterministic decoding or candidate-set size. These results establish positional bias as a measurable, reproducible property of prompt-mediated recommendation pipelines in the evaluated setting and motivate routine auditing and governance controls (e.g., order randomization, standardized prompting) for deployed LLM-based RS. Source code is publicly released.

1 Introduction

In prompt-mediated LLM recommenders, candidate order can influence recommendations independently of relevance, but this behavior has not been formally measured in this setting. The paper develops statistical tests and metrics and finds significant positional bias across evaluated models.

1 Introduction

  • The study measures positional bias in top-1, single-turn LLM recommenders rather than proposing mitigation strategies.Its questions address whether position affects selection, whether that dependence is statistically significant, and how its magnitude varies across models.Source: §1 · p. 2
  • POB-ICM tests whether candidate position and output selection are independent using contingency-matrix analysis and a query-level permutation test.The query-level test accounts for dependence among permutations of the same query.Source: §1 · p. 2
  • POB-UNBP uses a uniformity null model to reduce sample requirements and accommodate stochastic decoding.The paper also introduces permutation sensitivity and positional bias metrics to distinguish decision changes under permutation from systematic position preference.Source: §1 · pp. 2–3
  • Strong positional bias measures cases where position moves recommendations outside query-dependent sets of comparably relevant candidates.Counterfactual analyses report that this component is not explained by item identifiers and, for six of eight models, not by stochastic decoding alone; separate formal detection is future work.Source: §1 · p. 3
  • Robustness analyses across candidate-set size and decoding temperature identify positional bias as structural and model-dependent in the evaluated setting.The testbed uses a real product catalog, a fixed prompt template, and template-generated queries.Source: §1 · p. 3
  • All evaluated models reject positional independence and uniformity under permutation-based testing, with structured position-specific selection probabilities.The study focuses on a controlled RAG product-recommendation setting.Source: §1 · p. 3

2 Related Work

Prior work documents position-related effects across language-model tasks, but existing approaches do not provide hypothesis tests tailored to top-1 recommendation, where multiple candidates may be relevant. The paper introduces complementary methods for statistically detecting and quantifying this bias.

2 Related Work

  • Earlier recommendation studies report degraded performance when relevant items appear later, but do not isolate position from retrieval quality or item relevance.Top-1 recommenders expose only the selected item, so relevance assigned to other candidates must be inferred indirectly.Source: §2 · p. 4
  • Existing LLM positional-bias studies rely on performance or scalar metrics rather than formal hypothesis tests, and MCQ methods assume a unique correct answer.Top-1 recommendation may have multiple plausibly relevant candidates, complicating direct transfer from question answering.Source: §2 · p. 4
  • POB-ICM tests positional dependence through contingency-matrix independence, while POB-UNBP uses a uniformity-based null model for sample-efficient testing.The methods are tailored to controlled permutations in single-turn top-1 LLM recommendation.Source: §2 · p. 5
  • The paper distinguishes soft bias among comparably relevant candidates from strong bias that shifts recommendations toward clearly less relevant items.It quantifies the strong component alongside overall positional bias.Source: §2 · p. 5

3 Methodology for positional bias detection in LLM-based RS

The methodology uses randomized candidate order to test whether recommendation choices depend on prompt position. It pairs an interpretable contingency-based test with a more sample-efficient uniformity-based test that supports stochastic outputs.

3.1 An implementation of an LLM-based RS for positional bias analysis

  • The RAG testbed retrieves top-k products by semantic similarity, places their titles and descriptions with the query in a fixed prompt, and asks an LLM to select one.The implementation uses a pre-trained embedding model and cosine distance over product-title embeddings.Source: §3.1 · p. 5

3.2 Mathematical notation and probability distributions of the LLM-based RS

  • Random permutations define the evaluation: R(X, σ) records the selected prompt position for query and candidate instance X under permutation σ.Across queries and random permutations, absence of positional bias implies equal selection probability 1/k for every prompt position.Source: §3 · p. 5, §3.2 · p. 6

3.3 POB-ICM: POsitional Bias detection methodology based on the Independency of the Contingency Matrix

  • POB-ICM tests conditional independence between the recommended item's original index and the applied permutation using a contingency matrix.Its query-level permutation calibration shuffles outcomes only among permutations of the same query to account for within-query dependence.Source: §3.3 · p. 7
  • POB-ICM's matrix can reveal whether products moved to particular prompt positions are selected more or less often.The full contingency matrix can be costly to construct and visually large, motivating a simplified position-based view.Source: §3.3 · p. 7, §3.4 · p. 8

3.4 POB-UNBP: Positional Bias detection methodology based on Uniformity assuming Non-Biased Positions

  • POB-UNBP derives a uniform position distribution under no positional bias and uses deviations from uniformity for sample-efficient detection.The method generalizes to stochastic outputs, allowing small tolerated positional effects while retaining a uniformity-based test.Source: §3.4 · p. 8, §3.4.1 · p. 8, §3.4.2 · p. 9

3.4.1 Zero-temperature setting

  • In the zero-temperature setting, absence of positional bias means each query yields the same recommended item under every permutation.Uniform random shuffling then places that fixed item uniformly across prompt positions.Source: §3.4.1 · p. 8

3.4.2 Stochastic setting

  • For stochastic decoding, ε-absence requires recommendations across permutations to remain sufficiently similar, and implies total variation distance from uniformity is at most ε.The test estimates position-level deviations from uniformity across independently sampled query instances and can use one permutation per instance.Source: §3.4.2 · pp. 8–9
  • Increasing permutations per query reduces only within-query variance; between-query variance remains irreducible and limits gains from more permutations.The experiments use 20 permutations per query, while the method is valid with one or more.Source: §3.4.2 · p. 10

4 Empirical results and analysis of positional bias

This section evaluates positional independence and uniformity with POB-ICM and POB-UNBP, and examines position preferences and metrics for recommendation instability and strong positional bias. It analyzes how candidate-set size and decoding temperature relate to positional bias, noting that strong-bias estimates include a noise floor and are upper bounds.

4.1 Evaluation dataset

  • The dataset combines 2,000 template-generated queries with 50,000 sampled products; each query retrieves five candidates, evaluated under 20 permutations for 40,000 instances.The queries were sampled from 56 templates, and the study also built analogous 40,000-instance datasets for candidate-set sizes k = 10, 15, and 20.Source: §4.1 · pp. 10–11

4.2 Positional bias metrics

  • Permutation sensitivity measures recommendation changes under reordering, positional bias measures preference against uniform position selection, and strong positional bias measures recommendations outside a relevance set.The relevance set is the smallest set of most frequently recommended items accounting for a calibrated fraction of recommendations; its threshold is calibrated across instances, and sPB uses five-fold cross-validation. Positional bias includes both soft bias among comparably relevant items and strong bias toward less relevant items.Source: §4.2 · pp. 12–13

4.3 Experimental setup and implementation details

  • The setup retrieved candidates using the BAAI embedding model and evaluated deterministic prompts across LLaMA 3.1 [8], LLaMA 4 Scout, GPT-OSS [16], and DeepSeek R1.Parsing errors were filtered before statistical testing; restricting POB-UNBP to queries without discarded generations left its conclusions unchanged.Source: §4.3 · p. 13

4.4 Results and analysis

  • Both POB-ICM and POB-UNBP detect positional bias in every evaluated model; POB-UNBP confirms it with fewer samples than POB-ICM.POB-ICM rejects independence between candidate position and selected output at p < 0.05; POB-UNBP confirms non-uniformity across the evaluated models.Source: §4.4 · p. 13, Table 2 · p. 14, §4.4.1 · p. 14

4.4.1 Results and visualization of POB-ICM methodology

  • In POB-ICM visualizations, LLaMA 3.1 8B tends toward later prompt items, while GPT-OSS and LLaMA 3.1 70B tend toward earlier items.The patterns are consistent across exhaustive and sampled configurations; DeepSeek R1 8B and 32B show comparatively weaker bias. Differences between matrix rows reflect item relevance, whereas positional bias appears as differences between columns.Source: §4.4.1 · pp. 14–15, Figure 5 · p. 16

4.4.2 Results on POB-UNBP methodology

  • Across candidate-set sizes, DeepSeek R1 variants and LLaMA 3.1 8B favor middle over first positions, while GPT-OSS and LLaMA 3.1 70B favor first positions; LLaMA 4 Scout increasingly favors later positions as k grows.The preferred relative positions of DeepSeek R1 variants and LLaMA 3.1 8B also shift with candidate-set size.Source: §4.4.2 · pp. 16–17
  • At t = 0.0, 0.2, 0.4, and 0.6, every model remains significant at p < 0.05, and position-selection patterns stay stable, varying primarily by model identity.GPT-OSS and LLaMA 3.1 70B favor first positions; LLaMA 3.1 8B and DeepSeek R1 8B and 32B favor later positions; DeepSeek R1 7B concentrates bias on the second position, and LLaMA 4 Scout avoids the first.Source: §4.4.2 · p. 17

4.4.3 Positional bias quantification

  • For some queries, one item dominates across permutations; for others, selections split among comparably relevant items, motivating a query-dependent relevance set to avoid overstating strong bias.The latter pattern reflects soft rather than strong positional bias when changes occur among similarly relevant candidates.Source: §4.4.3 · pp. 17–18, Figure 7 · p. 20
  • 34.0%→58.1%: DeepSeek R1 7B’s PS rose from k = 5 to k = 20, while LLaMA 4 Scout’s PB rose from 2.9% to 33.8%.PS increases monotonically with k for every model, and PB generally increases, with a slight DeepSeek R1 7B decrease from k = 15 to k = 20. GPT-OSS (20B and 120B) and LLaMA 3.1 70B show more modest PB growth.Source: §4.4.3 · p. 18, Table 6 · p. 20
  • 3.2%→36.6%: LLaMA 4 Scout’s sPB increased from k = 5 to k = 20, showing that larger candidate sets amplify strong position-driven changes.sPB increased monotonically for every model; DeepSeek R1 8B rose from 6.7% to 20.6%. The growth indicates that a larger share of permutation-induced changes involve items outside the estimated relevance set.Source: §4.4.3 · p. 19, Table 6 · p. 20
  • 0.9%–9.1%: the estimated sPB noise floor at fixed order makes absolute sPB values upper bounds on strong positional bias.For six of eight models, permutations added 0.3–1.1 percentage points to this floor; trends across candidate-set sizes and models are more informative than absolute sPB values.Source: §4.4.3 · p. 20
  • 4.2%→3.0%: LLaMA 3.1 8B’s PB decreased as temperature rose from 0.0 to 0.6, and PB showed no consistent upward temperature trend across models.PS increased only mildly for LLaMA 3.1 8B, from 30.4% to 33.0%, while the modest PS and sPB increases were attributed to additional decoding stochasticity rather than stronger positional bias.Source: §4.4.3 · pp. 19–20, Table 7 · p. 21

5 Conclusion and future work

In top-1, single-turn recommendation, candidate order significantly influences model selections, with bias varying across models and generally increasing as more items enter the prompt. The paper frames its contribution as a detection and measurement framework, while identifying deployment controls and broader evaluations as practical and future directions.

5 Conclusion and future work

  • All evaluated models reject positional independence and uniformity; bias varies across models and generally increases as more candidates are presented.Strong positional bias persists after controlling for item identifiers and, for six of eight models, stochastic decoding.Source: §5 · p. 21
  • The evidence is scoped to top-1 product recommendation using template-generated queries, a real catalog, and one fixed prompt; other prompts and domains may yield different bias patterns.Top-1 makes relevance difficult to infer because only one selected item is observed.Source: §5 · p. 21
  • Prompt order is a first-class design variable because it can change recommendations without corresponding differences in item relevance.The paper connects order-sensitive behavior to reliability, offline evaluation, A/B testing, auditability, and accountability.Source: §5 · p. 21
  • Recommended controls include randomized or counterbalanced ordering, standardized prompts, and permutation-based tests in routine validation and monitoring.The paper also discusses mitigation through order randomization, ensemble prompting, or post-generation re-ranking.Source: §5 · p. 22
  • The framework applies to black-box systems that select a candidate by position, independent of internal architecture, retrieval mechanism, or turn count.Multi-turn use is possible by evaluating each turn as a separate top-1 instance, but transfer of the empirical findings requires new experiments.Source: §5 · p. 22
  • Top-N recommendation and reranking require extending the bias concept and adapting the null models to ranked outputs.The paper leaves these adaptations for future work.Source: §5 · p. 22
  • The study uses one fixed prompt template, and measured bias magnitude and pattern may depend on prompt design.The authors say this does not affect the validity of the detection methodologies.Source: §5 · p. 22
  • Future work includes detecting soft and strong bias independently, testing richer queries and other domains, incorporating quality and utility measures, and developing mitigation methods.The paper also identifies interactions with other structural biases and production-scale diagnostics as open directions.Source: §5 · pp. 22–23

A Proof of Theorem 1 and Proposition 2

The appendix proves a total-variation bound under the uniformity null model and establishes the corresponding confidence-bound result. Its proof combines permutation averaging with concentration arguments and then extends the bound across instances.

A Proof of Theorem 1 and Proposition 2

  • Under ε-absence of positional bias, the total variation distance between the selection distribution and uniformity is at most ε, both conditionally on each instance and unconditionally.The unconditional result follows by convexity of total variation and Jensen’s inequality.Source: A Proof of Theorem 1 and Proposition 2 · p. 25
  • For a fixed instance, the proof averages selection probabilities over uniformly sampled permutations and bounds their deviation from the uniform distribution using the definition of positional bias.The argument fixes a reference permutation and uses the uniform distribution of each item’s position.Source: A Proof of Theorem 1 and Proposition 2 · p. 25
  • Proposition 2 introduces a bound L computed from n independent instances, with its stated conditions including k ≥ 2, ℓ ≥ 1, and α ∈ (0, 1).The supplied passage does not include the proposition’s concluding inequality.Source: A Proof of Theorem 1 and Proposition 2 · p. 25
  • The proof controls each coordinate’s upper confidence bound using bounded i.i.d. variables, separating zero-variance and positive-variance cases before applying asymptotic normality and a union bound.When all coordinate bounds hold, monotonicity yields the total-variation bound.Source: A Proof of Theorem 1 and Proposition 2 · p. 26

B Simulation study: Type-I error and statistical power

Synthetic simulations compare the tests’ calibration, power, and query requirements under known injected positional bias. POB-UNBP remains calibrated and is substantially more sample-efficient than POB-ICM, which loses power at larger candidate-set sizes.

B Simulation study: Type-I error and statistical power

  • The simulation injects strong bias by selecting the first-position item with probability p_bias; otherwise, it selects uniformly from a randomly drawn non-empty relevance set.p_bias = 0 is the no-bias null, and experiments cover k ∈ {5, 10, 15, 20}.Source: B Simulation study: Type-I error and… · p. 26
  • POB-UNBP’s Type-I error is 0.039–0.053 across k, while POB-ICM is conservative at k ≥ 15 with error 0.016–0.020.The authors attribute POB-ICM’s conservatism to sparse contingency-matrix columns producing nearly constant chi-square statistics and ties.Source: B Simulation study: Type-I error and… · p. 26, Table 8 · p. 27
  • At k = 5 and nσ = 20, POB-UNBP reaches 80% power with 175 queries versus 1,068 for POB-ICM; POB-ICM never reaches that power for k ≥ 10.POB-UNBP requires 65 queries at k = 20, and the study’s 2,000-query, 20-permutation operating point affords high power for both tests.Source: Table 9 · p. 27, B Simulation study: Type-I error and… · p. 27
  • With 500 queries and 20 permutations, POB-UNBP power saturates by p_bias ≈ 0.02 at k = 5 and at smaller bias for larger k; POB-ICM needs roughly 0.02–0.06 at k = 5.POB-ICM fails to gain power for k ≥ 10.Source: B Simulation study: Type-I error and… · p. 26, Figure 8 · p. 27

C Separating stochastic decoding variability from positional bias

The analysis separates changes caused by permutations from those caused by stochastic decoding using matched evaluation configurations. Overall positional bias remains under temperature, and adding permutations increases strong positional bias for most evaluated models, with an important GPT-OSS qualification.

C Separating stochastic decoding variability from positional bias

  • The counterfactual design compares decoding-only, permutation-only, and combined variability over 20 evaluations per instance, then tests paired sPB differences between combined and decoding-only conditions.The paired Wilcoxon test excludes unchanged instances and applies Holm correction across eight models.Source: C Separating stochastic decoding… · pp. 27–28
  • Adding permutations to stochastic decoding significantly raises strong positional bias for six of eight models by 0.3–1.1 percentage points; GPT-OSS models are exceptions.Across configurations, sPB ranges from 0.9% to 10.2%; the table reports sPB, overall-bias lower bounds, relevance-set IoU, and adjusted comparison p-values.Source: Table 10 · p. 28, C Separating stochastic decoding… · p. 28
  • Overall positional bias persists under stochastic decoding: L is 0.8–4.7 percentage points with permutations alone and 1.0–4.9 with permutations plus temperature.These magnitudes are described as similar with and without temperature.Source: C Separating stochastic decoding… · p. 28
  • For GPT-OSS, the study finds no evidence of strong positional bias beyond noise, consistent with a first-position preference acting mainly as a tie-breaker among comparably relevant items.The comparison is qualified because the identity permutation couples retrieval rank to first prompt position.Source: C Separating stochastic decoding… · p. 28
  • Relevance sets overlap by 70%–80% across decoding-only and permutation configurations, with differences typically involving one item.Some differences reflect estimation noise because each relevance set uses only 20 evaluations.Source: C Separating stochastic decoding… · p. 28

D Separating item identifiers from positional bias

With randomly permuted identifiers, the analysis finds positional bias beyond identifier effects across all eight models, including strong bias that increases when identifiers and positions are decoupled.

D Separating item identifiers from positional bias

  • The top-5 analysis randomly assigns identifiers independently of query and item permutation, then maps returned labels back to prompt positions.It compares random labels only, permutations only, and both together.Source: D Separating item identifiers from… · p. 29
  • With fixed order, random labels alone produce sPB of 1.0% to 9.5%, comparable to the stochastic-decoding noise floor.The authors describe identifiers as a prompt perturbation similar to decoding noise.Source: D Separating item identifiers from… · p. 29
  • With random labels, POB-UNBP rejects uniformity for all eight models (p ≤3.5 · 10−4), so identifiers do not explain overall positional bias.The magnitude varies: L rises for five models, stays similar for two, and drops from 3.0 to 0.3 for LLaMA 3.1 8B.Source: D Separating item identifiers from… · p. 29
  • Adding permutations to random labels significantly increases sPB for all eight models by 0.5 to 1.3 percentage points.Compared with positional labels, sPB increases significantly for six models and is unchanged for LLaMA 4 Scout and LLaMA 3.1 70B.Source: D Separating item identifiers from… · p. 29
  • For GPT-OSS models, decoupling identifiers raises sPB from 1.0% to 1.5% and 2.1% to 3.0%, alongside increases in L from 3.1 to 4.3 and 4.7 to 7.5 percentage points.The authors suggest these models may rely more on item position when identifiers no longer follow prompt order.Source: D Separating item identifiers from… · p. 29
  • The random-label-only configuration keeps the identity permutation, so the retriever’s first-ranked product always occupies the first prompt position.Source: D Separating item identifiers from… · p. 30

E Confidence intervals for permutation sensitivity and strong positional bias

Query-level bootstrap intervals assess permutation sensitivity and strong positional bias across candidate-set sizes and decoding temperatures. The sensitivity to candidate-set size is robust, while strong positional bias is essentially stable across temperatures.

E Confidence intervals for permutation sensitivity and strong positional bias

F Effect of retrieval score dispersion on strong positional bias

The analysis tests whether retrieval-score dispersion predicts query-level strong positional bias. Its estimated association is small and does not explain the detected bias.

F Effect of retrieval score dispersion on strong positional bias

  • The study regresses each query’s percentage of valid recommendations outside its relevance set on standardized retrieval-score dispersion, weighting by valid evaluations.It tests zero slopes using two-sided Wald tests with HC1 standard errors and Holm correction across 32 tests per table.Source: F Effect of retrieval score dispersion… · p. 32
  • Across every configuration, the slope is negative: less-separated candidates show slightly more strong positional bias, by at most about 1 percentage point per standard deviation.The effect is significant mainly for k ≥10; at k = 5 it is significant for only a few models and shows no temperature trend.Source: F Effect of retrieval score dispersion… · p. 32
  • Retrieval-score dispersion does not explain detected positional bias and only marginally modulates its strong component.Source: F Effect of retrieval score dispersion… · p. 32

G Sensitivity to discarded generations

A complete-case analysis checks whether discarding failed generations alters POB-UNBP estimates. The tests continue to reject positional independence, and the estimated bias remains stable overall.

G Sensitivity to discarded generations

Loading 2610.37962v1…