PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Position Bias in Dense Retrievers Built In-or Learned from Data?

Daegon Yu, SeungYoon Han, Woomyoung Park

arXiv:2605.26578v1cs.IR

TL;DR

Dense retrievers exhibit positional bias, and the paper asks whether fine-tuning data can shape that bias beyond architecture and pretraining. It constructs position-targeted datasets and fine-tunes eight diverse models, finding that skewed data induces matching retrieval preferences while balanced training reduces positional sensitivity.

  • Problem

    The paper investigates whether fine-tuning data's positional distribution can change retrieval-level position bias beyond tendencies induced by architecture and pretraining.

  • Method

    The authors construct position-targeted datasets and fine-tune eight architecturally diverse pretrained models under skewed and balanced evidence-position distributions.

  • Results

    Across eight models, skewed training distributions favor corresponding evidence positions, while balanced training reduces positional sensitivity with competitive retrieval performance.

  • Takeaways & Limitations

    Training-position distribution is a controllable factor for retrieval-level position bias, making balanced data curation a supported mitigation strategy in this setting.

  • Takeaways & Limitations

    The synthetic English-Wikipedia setting leaves physical position partially entangled with segment content, discourse role, query semantics, and difficulty.

Abstract

from arXiv · show

Dense retrievers exhibit positional bias, favoring documents whose query-relevant information appears near the beginning and degrading retrieval performance when the information appears later. While prior work on positional bias in dense retrievers has largely focused on architectural explanations, we study how the positional distribution of evidence in training data affects retrieval-level bias direction. To test this, we construct synthetic position-targeted training sets in which query-relevant evidence appears at the beginning, middle, or end of documents, and fine-tune eight architecturally diverse pretrained models under position-skewed and balanced training distributions. At the ranking level, we observe a strong directional pattern across the examined models: skewed training distributions favor evidence at the corresponding positions. Position-balanced training reduces positional sensitivity by 57--87\% on position-aware benchmarks, with competitive mean retrieval performance in our controlled setting. Representation-level analyses further suggest that fine-tuning often reshapes learned positional preferences, although pre-existing architectural or pretraining-specific tendencies persist in some models. These results identify training-position distribution as a major controllable factor in retrieval-level position bias and suggest balanced data curation as a practical mitigation strategy.

1 Introduction

Dense retrievers show position bias, but this study tests whether fine-tuning data can shape its direction. Controlled experiments find that training-position distributions determine retrieval preferences, while balanced training reduces sensitivity.

  • Position bias can cause dense retrievers to miss relevant information appearing later in documents, threatening downstream retrieval-based tasks.
  • The study asks whether fine-tuning data can change retrieval-level position bias beyond tendencies induced by architecture and pretraining.
  • The authors construct position-controlled datasets and fine-tune eight architecturally diverse pretrained models with evidence targeted to document beginnings, middles, or ends.
  • Training-position distributions determine bias direction across models: begin-, middle-, and end-skewed data produce corresponding retrieval preferences.
  • Position-balanced training reduces positional sensitivity while preserving competitive retrieval performance, identifying data curation as a potential mitigation.

2 Related Work

Prior research establishes that positional bias is widespread across dense retrievers and model configurations, but architectural explanations do not fully account for it. Training data has been implicated, yet its causal role remains insufficiently isolated.

  • Dense embedding and ColBERT-style retrievers often degrade when evidence appears later, whereas BM25 and cross-encoder rerankers remain comparatively robust.
  • Position sensitivity appears across model sizes, vector dimensions, attention mechanisms, pooling strategies, and positional encodings, without clear architectural correlation.
  • Architectural studies link primacy bias to pooling-token attention and causal attention, but the generality of these explanations across dense-retrieval architectures is unresolved.
  • Prior work associates training data with position bias through unsupervised pretraining and early-position-heavy MS MARCO fine-tuning, motivating direct manipulation of training distributions.

3 Method

The method builds position-targeted datasets, verifies evidence placement with multiple rerankers, and constructs matched fine-tuning configurations. Sampling controls document length and training scale so positional distributions are the main experimental variable.

  • The data pipeline prepares length-stratified documents, generates position-targeted queries, and verifies target positions with multiple rerankers.
  • English Wikipedia articles are divided into beginning, middle, and end segments within five character-count length bins.
  • Queries target each segment through persona-conditioned generation, with the same document paired separately with beginning-, middle-, and end-targeted queries.
  • Candidates are retained only when every reranker scores the intended segment at least δ higher than the strongest non-target segment, using δ = 0.3.
  • 481,236 candidate examples survive verification, after which downsampling balances length-position cells before controlled training.
  • The experiments compare four begin:middle:end training ratios across eight models, holding training scale and document-length distributions approximately constant.

4 Experimental Setups

The evaluation spans architecturally diverse retrievers and position-aware as well as conventional retrieval benchmarks. Positional sensitivity is summarized with PSI alongside mean retrieval performance.

  • Base Models: The eight base models include encoder and decoder architectures, multiple positional encodings, and different pooling strategies.
  • Training Details: All models are fine-tuned as bi-encoder retrievers with InfoNCE loss and chunk-aware negatives, while hyperparameters remain fixed across configurations within each model.
  • Evaluation: Position-aware evaluation uses SQUAD-POSQ, FINEWEB-POSQ, and POSIR, with longer benchmarks restricted to models having sufficient context length.
  • Evaluation: Four evidence-annotated BEIR datasets additionally support analysis of evidence position under conventional retrieval evaluation.
  • Evaluation: PSI summarizes positional sensitivity from nDCG@10 across beginning, middle, and end subsets; zero indicates robustness, while higher values indicate greater sensitivity.

5 Experimental Results

Skewed fine-tuning distributions produce retrieval preferences for corresponding evidence positions across models, while balanced training reduces positional sensitivity and maintains competitive retrieval performance. Standard BEIR scores also vary with benchmark evidence-location skew.

  • Begin-, middle-, and end-trained retrievers favor early, middle, and later evidence, respectively, consistently across all eight base models.Uniformly trained retrievers have flatter position-wise curves rather than a comparable single-position preference.
  • 57–87%: uniform training reduces Position Sensitivity Index relative to the worst skewed configuration for every model on SQuAD-PosQ.The same reduction pattern holds on FineWeb-PosQ for the reported ModernBERT and Qwen3 models.
  • Uniform training produces more stable retrieval across evidence positions without merely shifting the bias to another location.The uniform configuration is the least sensitive on the position-aware benchmarks.
  • Uniform training achieves the highest mean nDCG@10 for five of eight SQuAD-PosQ models and all three evaluated FineWeb-PosQ models.For the remaining SQuAD-PosQ models, its gap to the best skewed configuration is 0.004–0.007.
  • 0.333: begin-trained models have the highest average BEIR nDCG@10 across four subsets, followed by MU at 0.297, MM at 0.212, and ME at 0.193.The begin-trained advantage is largest on early-skewed FEVER and HotpotQA, reverses on SciFact, and is nearly zero on Climate-FEVER.
  • BEIR gains can reflect benchmark-specific evidence-location skew rather than robustness to evidence appearing elsewhere.This follows the reported alignment between subset evidence distributions and training-induced positional priors.

6 Analyses

Analyses show that fine-tuning changes positional preferences in query-document similarities and document representations, while the directional effect persists across pooling strategies. Uniform training produces flatter similarity patterns and more position-balanced retrieval.

  • The highest query-document similarity position follows the fine-tuning distribution for ModernBERT-base and Qwen3-0.6B.MB peaks at position 1, ME at position 9, MM in the middle, while uniform training substantially narrows peak-to-lowest gaps.
  • Fine-tuning shifts document-embedding similarity profiles toward the training distribution, beyond query-document matching scores alone.Pretrained models retain mild initial positional tendencies, but retrieval fine-tuning redirects them toward emphasized evidence positions.
  • Across CLS, mean, max, and last-token pooling, skewed training favors the corresponding positions and uniform training yields a more position-balanced pattern.The pooling ablation uses ModernBERT-base on both SQuAD-PosQ and FineWeb-PosQ.
  • Changing fine-tuning position distribution has a larger observed effect on retrieval-level bias direction than changing pooling method in the controlled ModernBERT-base ablation.

7 Conclusion

Across eight architecturally diverse retrievers, skewed fine-tuning induces corresponding ranking-level positional preferences, while balanced training reduces sensitivity with competitive retrieval performance. Representation analyses suggest that fine-tuning can redirect document embeddings toward emphasized evidence positions, although model-specific tendencies remain.

  • Eight architecturally diverse retrievers develop ranking-level positional preferences aligned with skewed fine-tuning distributions.
  • 57–87%: position-balanced training reduces positional sensitivity on controlled position-aware benchmarks while maintaining competitive retrieval performance.
  • Fine-tuning can shift document embeddings toward emphasized evidence positions, while model-specific positional tendencies remain.

Limitations

The study's evidence comes from a synthetic, position-targeted English Wikipedia setting, so physical position may remain entangled with content and query-related factors. Additional methodological constraints include model-based rather than human annotation, single-seed controlled experiments, and limited downstream evaluation.

  • Synthetic position-targeted examples may confound physical position with segment content, discourse role, query semantics, and difficulty.Training scale and document-length distributions are matched across configurations, but target segments and generated queries differ.
  • The results support training-position distributions as an influence on retrieval-level bias, not physical position alone as its sole cause.
  • The retained pool is not human-annotated, so residual labeling errors or verifier-induced biases may remain.
  • The experiments use a controlled single-seed setup without hard-negative mining, early stopping, or extensive hyperparameter sweeps.Small mean nDCG differences should therefore be treated as point estimates.
  • Evaluation covers position-aware benchmarks and four evidence-annotated BEIR subsets, but not end-to-end RAG or production retrieval systems.Future work is identified for human-validated, multilingual, domain-specific, and downstream settings.

Ethics Statement

The study constructs synthetic position-targeted retrieval data from English Wikipedia using LLM-generated queries, without collecting private user data or conducting human-subject experiments. Because Wikipedia contains material about real people, organizations, and sensitive topics, derived examples may retain public names or sensitive or offensive content.

  • The dataset is built from English Wikipedia and LLM-generated queries for retrieval-robustness research.
  • The study does not collect private user data, conduct human-subject experiments, or infer protected attributes.
  • Wikipedia-derived examples may include public names or sensitive or offensive content from source articles.
  • A shared configuration is selected per document and inherited across position-conditioned query generation stages.The configuration includes persona, difficulty, and query length, and is reused across positional queries.
  • Generated queries are intended to be answerable only from their target segments, but the positional constraint is not guaranteed by the generator.Candidates are subsequently validated through a multi-reranker filtering pipeline.
  • The retained candidate pool is used as a source for final training sets, which are downsampled within length-position cells.

B.1 Consensus Margin for Threshold Analysis

The consensus-margin analysis retains candidates when three rerankers agree that the intended target segment outranks both non-target segments. The δ = 0.3 pool is conservative and high-confidence but position-skewed, so final configurations use controlled sampling; an independent LLM audit validates segment exclusivity.

  • B.1 Consensus Margin for Threshold Analysis: The consensus margin is the smallest target-versus-non-target score gap among three rerankers.
  • B.1 Consensus Margin for Threshold Analysis: A candidate passes threshold δ exactly when m_cons ≥ δ; m_cons ≥ 0 means all rerankers rank the target above both non-target segments.Larger thresholds require stronger agreement that the query is grounded in the intended target segment.
  • B.1 Consensus Margin for Threshold Analysis: The final training sets use the retained pool obtained with margin threshold δ = 0.3.
  • B.1 Consensus Margin for Threshold Analysis: 481,236 candidates remain at δ = 0.3, including 62,904 middle-targeted and 82,682 end-targeted candidates.The pool contains enough examples in every target position for controlled sampling despite increasing begin-skew at stricter thresholds.
  • B.1 Consensus Margin for Threshold Analysis: The δ = 0.3 pool is conservative and higher-confidence than lower-margin strata, but it is neither position-balanced nor length-neutral.It is therefore used as a source pool rather than the final training distribution.
  • B.3 Segment-Wise LLM Audit: The held-out LLM audit independently evaluates each query against all three segments using binary answerability judgments.The audit is post-hoc validation, not part of training-set construction.
  • B.3 Segment-Wise LLM Audit: Exclusive requires the target to be answer-containing while both non-target segments are answer-absent.The exclusive rate increases monotonically across higher-margin strata, with m_cons ≥ 0.3 highest.
  • B.4 Final Sampling from the δ = 0.3 Retained Pool: Final configurations downsample within length-position cells to prevent retained-pool counts from becoming confounding factors.The smallest retained cell contains 8,189 examples and sets the common per-bin budget for concentrated configurations.

E Additional Experimental Results: PosIR

PosIR reproduces the paper's central directional pattern: position-skewed training favors corresponding evidence locations, whereas uniform training yields flatter position-wise performance. On the evaluated long-context models, uniform training also combines the highest mean nDCG@10 with the lowest PSI.

  • E Additional Experimental Results: PosIR: PosIR shows begin-trained retrievers favoring earlier evidence, mid-trained retrievers peaking near the middle, and end-trained retrievers improving toward later evidence.
  • E Additional Experimental Results: PosIR: Uniformly trained retrievers produce flatter position-wise curves, indicating lower sensitivity to physical evidence location.
  • E Additional Experimental Results: PosIR: 0.411, 0.423, and 0.450 are the mean nDCG@10 values for uniformly trained ModernBERT-base, ModernBERT-large, and Qwen3-0.6B, respectively.Uniform training also reduces PSI relative to the worst skewed configuration by 73.5%, 75.4%, and 55.8% for those models.
  • E.1 Mirror-Reversal Diagnostic on PosIR: The mirror-reversal diagnostic compares the same origin groups before and after reversing five document segments.Positive Δ_rev means back-origin evidence performs better after moving to the front; negative Δ_rev means front-origin evidence performs better after moving to the back.

F Evidence-Moving Analysis Full Results

Fine-tuning generally aligns embedding-level evidence preferences with the target position distribution, while uniform training compresses positional differences. Some architectures retain residual or incomplete alignment, but these deviations do not alter the ranking-level pattern across models.

  • Five of eight models align embedding preferences directionally with the fine-tuning distribution.Begin-trained models peak at p1, end-trained models at p9 or p10, and mid-trained models between p3 and p5.
  • Uniform training yields the smallest or near-smallest Range for all eight models.The sole exception is BLOOM-560M, where uniform training has Range 5.4 versus 4.7 for mid training.
  • Uniform training compresses peak-to-lowest cosine differences and weakens position-specific preference.For BLOOM-560M, both mid- and uniform-trained ranges remain far below the begin- and end-trained ranges.
  • GPT-2-medium, BERT-base, and TinyLlama-NoPE show model-specific deviations from clean directional alignment.GPT-2-medium retains a late preference across concentrated settings; BERT-base incompletely follows end training; TinyLlama-NoPE fails to shift under mid training.
  • These embedding-level deviations do not change the ranking-level conclusion across all eight models.Position-skewed fine-tuning induces retrieval behavior aligned with the corresponding training-position distribution, although preference strength and exact location vary by architecture.
Loading 2605.26578v1…