PaperlayerView exact revision on arXiv

Source-linked AI summary

Your Prompt Should Do More: Effects of Retrieval Instructions in Embedding Models

Amanda Myntti, Jenna Kanerva, Veronika Laippala, Filip Ginter

arXiv:2610.10508v1cs.CL

TL;DR

Standard evaluation often isolates retrieval tasks, leaving limited evidence about how instructions alter query representations when candidates include semantically similar distractors. The paper analyzes instruction effects, evaluates query-side distractors, and fine-tunes with query-side negatives. Models can fail to follow instructions under distractors, while this fine-tuning substantially improves robustness with little change to standard retrieval.

  • Problem

    Task-isolated retrieval benchmarks provide limited insight into how instructions transform query representations when candidate pools include semantically related texts that do not satisfy the task.

    Source: §1 · p. 1

  • Method

    The study analyzes instruction effects on retrieval representations, evaluates semantically similar query-side distractors, and fine-tunes with query-side negatives.

    Source: §5 · p. 7, §6 · p. 9

  • Results

    Fine-tuning with query-side negatives substantially improves robustness to paraphrase distractors, while standard retrieval performance remains close to baseline.

    Source: §1 · p. 1, §5.1 · p. 8

  • Takeaways & Limitations

    Task-isolated evaluation may overestimate robustness, and adding query-side negatives can improve instruction following under distractors.

    Source: §1 · p. 1, §6 · p. 9

    The study covers retrieval and bitext mining and eight prominent models, so generalization beyond those tasks and models is not guaranteed.

    Source: §6 · p. 10

Abstract

from arXiv · show

Prompted embedding models have recently received increasing attention, particularly for retrieval, where detailed retrieval instructions are provided as part of the retrieval prompt. Several new datasets and studies have examined this setting, showing that the current embedding models often struggle to follow such instructions reliably. In this paper, we study the mechanism of how instructions actually affect the representations of retrieval queries in asymmetric retrieval tasks. We show that models can fail to follow even simple task instructions when query-side distractors are included in the evaluation. We hypothesize that this behavior is driven by the training setup of current embedding models and their evaluation, and show that fine-tuning with added query-side distractors leads to substantial improvements, with minimal effect on other tasks.

1 INTRODUCTION

Standard retrieval benchmarks evaluate tasks in isolation, leaving limited insight into how instructions transform query embeddings when semantically related non-matches are present. In controlled evaluation, semantic similarity can dominate task instructions, while fine-tuning with query-side negatives substantially improves robustness.

1 INTRODUCTION

  • Task-isolated benchmarks rarely include semantically related texts that fail the task instruction, limiting insight into how instructions change query embeddings and retrieval behavior.The paper argues that this mismatch may obscure challenges encountered in practical large-scale retrieval.Source: §1 · p. 1
  • With query-side distractors, semantic similarity can dominate the required task relation, and models may fail to follow even simple instructions.The authors hypothesize that this weakness may reflect common contrastive training practices that lack query-side negatives.Source: §1 · p. 1
  • Fine-tuning with query-side negatives substantially improves robustness to distractors, with minimal effect on other tasks.The paper reports that the training change addresses inadequate instruction following.Source: §1 · p. 1

2 RELATED WORK

Prior work establishes contrastive bi-encoder models and asymmetric query instructions, while instruction-following benchmarks and representation studies leave systematic task-level effects unclear. This paper measures instruction-induced query shifts and tests query-side negatives as a way to improve instruction following.

2 RELATED WORK

  • Modern embedding models commonly use bi-encoder architectures trained with contrastive objectives (Gao et al., 2021), alongside hard-negative mining and synthetic negatives.Asymmetric task instructions are prepended to queries but not targets, allowing one model to support multiple retrieval tasks (Wang et al., 2024b).Source: §2 · p. 2
  • Instruction-following retrieval datasets report that models often struggle, but their complex, query-specific instructions are not designed to measure systematic task-level effects (Oh et al., 2024; Weller et al., 2025a;b; Zhou et al., 2025; Song et al., 2025; Weller et al., 2025c; Zhuang et al., 2025).Asai et al. (2023) instead introduce target annotations that let instructions define relevant documents.Source: §2 · p. 2
  • Prior studies examine embedding geometry and input effects, while Gonzalez-Gutierrez & Hovy (2025) find unrelated instructions can perform well and that instruction-induced representation changes remain poorly understood.The cited work notes that causes, mechanisms, and structure of these changes remain unknown.Source: §2 · p. 2
  • The paper analyzes roughly 50 diverse instructions with structural measures, then tests query-side hard negatives in fine-tuning and evaluation to strengthen instruction following.The authors identify the lack of query-side negatives in standard fine-tuning and isolated evaluation setups as a core weakness.Source: §2 · p. 2

3 METHODS

The study examines instruction effects across retrieval and bitext-mining tasks, using synthetic query-side distractors and structural measures of embedding movement. It then fine-tunes models with distractors while jointly retaining retrieval and semantic-similarity training.

3 METHODS

  • The experiments analyze instruction relevance and structural measures, test retrieval and bitext-mining differences with distractors, then fine-tune and inspect structural effects.The plan includes preserving performance on other tasks while improving instruction following.Source: §3 · p. 2

3.1 DATASETS, MODELS, AND INSTRUCTIONS

  • Tatoeba bitext mining, ARCChallenge retrieval, and SQuAD.v1 fine-tuning provide the study datasets (Tatoeba community, 2021; Clark et al., 2018; Xiao et al., 2024; Rajpurkar et al., 2016).The evaluation uses recall@1 for Tatoeba and ndcg@10 for ARCChallenge, matching MTEB metrics (Muennighoff et al., 2023; Enevoldsen et al., 2025).Source: §3.1 · pp. 2–3
  • For each query, a GPT-5.4-mini-generated paraphrase serves as a semantically similar query-side negative in evaluation and fine-tuning.Distractors test whether instructions shift embeddings from plain semantic matching toward the task-specific target.Source: §3.1 · p. 3
  • The experiments use eight embedding models and 47 instructions, or 51 for Tatoeba, including unrelated-task prompts to test how instruction relevance affects embeddings.The model set varies in benchmark performance and size; models above 8B parameters are excluded for computational reasons.Source: §3.1 · p. 3

3.2 STRUCTURAL STATISTICS

  • The structural statistics measure instruction effects through similarity improvement, query displacement, and movement angulation toward the correct answer.Similarity improvement compares query-to-answer cosine similarity with and without an instruction; displacement measures query movement, while angulation tests its direction relative to the answer.Source: §3.2 · pp. 3–4
  • Similarity improvement alone has a selectivity caveat: moving closer to the correct target may also mean moving toward incorrect targets.The three measures are intended to diagnose key characteristics of instruction effects.Source: §3.2 · p. 4

4 RESULTS

Task relevance offers only a distributional signal of retrieval performance, and structural effects of instructions differ across tasks. Adding semantically related distractors exposes substantial instruction-following failures, especially on ARCChallenge.

4.1 INSTRUCTIONS VS. EVALUATION PERFORMANCE

  • Relevant instructions offer a distributional performance signal, but unrelated prompts often rank highly and task relevance does not explain most score variation.Even a high-scoring ARCChallenge model places a keysmash instruction among its top results.Source: §4.1 · pp. 4–5, Table 2 · p. 4

4.2 SIMILARITY INCREASE, DISPLACEMENT AND ANGULATION

  • Relevant instructions are difficult to distinguish among high-scoring ARCChallenge prompts, but are more distinct on Tatoeba across structural measures.Across models, the separation between relevant and other instructions varies by task.Source: §4.2 · p. 5, Figure 1 · p. 5, Table 3 · p. 6
  • Across both datasets, lower displacement accompanies higher evaluation scores, while similarity-improvement and angulation relationships vary more.The displacement relationship is negative for both datasets; similarity improvement dominates on Tatoeba.Source: §4.2 · p. 5, Table 4 · p. 6

4.3 WHY DOES THIS HAPPEN?

  • The authors hypothesize that in-batch and target-side hard negatives may not move instructed queries toward answers enough when evaluation candidates are plausible targets.They suggest varied negatives during translation fine-tuning may improve language sensitivity and help explain Tatoeba results.Source: §4.3 · p. 6
  • With distractors, greater movement toward the target rather than the query helps retain performance.Figure 2 depicts relative normalized displacement among the query, distractor, and target for relevant instruction-query pairs.Source: §4.3 · p. 6, Figure 2 · p. 7
  • Distractors destroy performance for all instructions on ARCChallenge, while two Tatoeba models nearly retain it and their best-scoring instructions are relevant.The table compares maximal recall@1 without and with distractors; the prose reports the outcome.Source: §4.3 · p. 6, Table 5 · p. 7

5 DISTRACTOR FINE-TUNING

The paper fine-tunes retrieval models with query-side distractors, then evaluates distractor robustness and its effects on representation structure and other tasks. Fine-tuning substantially reduces the damage from paraphrase distractors while preserving standard retrieval and largely maintaining out-of-domain performance.

5 DISTRACTOR FINE-TUNING

  • Training uses generated query paraphrases as retrieval negatives and STS positives, alongside task-specific retrieval and STS examples.QA uses SQuAD.v1; bitext mining uses Tatoeba examples pooled across eight languages, with eight random target-side negatives per positive.Source: §5 · p. 7

5.1 EVALUATION AND RESULTS

  • Evaluation compares normal retrieval with random and semantically similar paraphrase distractors, adding one distractor per query.Paraphrases test whether models follow task instructions rather than relying on semantic similarity alone.Source: §5.1 · pp. 7–8
  • Fine-tuned models substantially reduce paraphrase-distractor damage while keeping normal retrieval close to baseline (Table 6).The baseline is nearly destroyed by paraphrase distractors, whereas random distractors cause only a moderate drop; improvements from fine-tuning occur mainly in the first 5,000 steps.Source: Table 6 · p. 8, §5.1 · p. 8

5.2 AFTER FINE-TUNING: WHAT HAPPENED?

  • After fine-tuning, relevant instructions increase similarity to answers more on SQuAD, but less on Tatoeba; displacement and angulation increase in both.On Tatoeba, the largest angulation increase occurs when the instruction explicitly names the language; relative-distance plots also show larger instruction effects and better performance retention with distractors.Source: §5.2 · pp. 8–9
  • Out-of-domain benchmark differences are small, and distractor fine-tuning does not destroy performance on other tasks.Figure 5 summarizes results by MTEB category; on ARCChallenge, paraphrase-distractor Recall@1 and Recall@5 improve slightly despite low baseline performance.Source: §5.2 · p. 9, Figure 5 · p. 9

6 DISCUSSION AND CONCLUSION

The discussion argues that standard evaluation may overestimate instruction sensitivity and proposes pooled, instruction-sensitive multi-task retrieval evaluation. The appendix reports how query-side distractors were generated.

6 DISCUSSION AND CONCLUSION

  • The authors argue that standard evaluation may overestimate genuine instruction sensitivity and call for pooled, instruction-sensitive multi-task retrieval evaluation.Their conclusion draws on analyses of instruction relevance, representation structure, distractor effects, and fine-tuning with selected negatives.Source: §6 · p. 9
  • The study focuses on retrieval and bitext mining, omits tasks such as clustering, and evaluates eight models, so generalization beyond those models is not guaranteed.The authors also note that synthetic distractors come from one generation model, without an ablation across generator choices.Source: §6 · p. 10

A GENERATING QUERY-SIDE DISTRACTORS

  • The experiments use one generated paraphrase per query, totaling 185,254 paraphrases at a reported cost of $10.22.The paraphrases were generated with GPT-5.4-mini through the Batch API.Source: A GENERATING QUERY-SIDE DISTRACTORS · p. 14
  • The appendix says the prompts used to generate synthetic distractors are provided in Table 7.The supplied passage identifies the table but does not give its prompt contents.Source: A GENERATING QUERY-SIDE DISTRACTORS · p. 14

B EMBEDDING MODELS

This appendix identifies the embedding models and their licenses, describes the instruction set and template, and introduces formulas for three structural metrics.

B EMBEDDING MODELS

  • Table 8 lists the study’s models, parameter counts, and embedding dimensions; the appendix records licenses including MIT, Apache-2.0, and Gemma.The listed examples include BAAI/bge-m3 under MIT and Qwen/Qwen3-Embedding-0.6B under Apache-2.0.Source: B EMBEDDING MODELS · pp. 14–15, Table 8 · p. 15

C INSTRUCTIONS

  • The instruction set includes task-specific, MTEB category, unrelated-task, vague, and selected LLM-generated instructions, using an “Instruct” and “Query” template.The appendix reports 47 instructions for retrieval and 51 for bitext mining, with additional bitext instructions specifying language.Source: C INSTRUCTIONS · p. 15

D FORMULAS FOR STRUCTURAL METRICS

  • The formulas define similarity improvement, displacement, and angulation using query, prompted-query, and target embeddings.The prose defines emb(·) as a text embedding and cos(·, ·) as cosine similarity.Source: D FORMULAS FOR STRUCTURAL METRICS · p. 15

E SUPPORTING RESULTS

Structural measures identify an instruction that performs best on the outlier model, but distractors still inhibit performance; fine-tuning also improves results for other relevant instructions.

E SUPPORTING RESULTS

  • “Retrieve the answer to the question” is the best-performing instruction across displacement, similarity improvement, and angulation for codefuse-ai/F2LLM-v2-4B on ndcg@10.The figure identifies this instruction as best in all three measures.Source: E SUPPORTING RESULTS · p. 15, Figure 8 · p. 17
  • Even the best instruction cannot fully overcome distractors for codefuse-ai/F2LLM-v2-4B on recall@1.The relative-distance plot reports that distractors still inhibit performance greatly.Source: E SUPPORTING RESULTS · p. 17, Figure 9 · p. 17
  • Fine-tuning gains are substantial for relevant instructions beyond the fine-tuning instructions, and all but one of the listed top improvers are task-relevant.The passage interprets this pattern as evidence that fine-tuning incentivized the model to react to instructions.Source: E SUPPORTING RESULTS · p. 17

F.1 SQUAD TRAINING DATA

  • SQuAD training reserves 2,009 examples for development and 75,018 for fine-tuning, while two instances per example produce 150,036 training examples.Negatives are sampled only across different answer paragraphs to avoid false negatives.Source: §F.1 · p. 17

F.2 TATOEBA TRAINING DATA

The appendix describes Tatoeba data filtering, fine-tuning settings, and task prompts, alongside reported results for instruction gains and ARCChallenge distractor evaluation.

F.2 TATOEBA TRAINING DATA

  • Table 10 lists the three instructions with the largest Recall@1 improvements during fine-tuning.Source: Table 10 · p. 18
  • ARCChallenge shows increased recall with paraphrase distractors but decreased rank-based NDCG; NDCG reveals distractor performance less strongly than recall@1.The table caption characterizes the ARCChallenge baseline as low.Source: Table 11 · p. 18
  • After filtering, most Tatoeba languages have 9,000 training and 1,000 development pairs; Vietnamese has 6,913 training and 1,000 development pairs.Filtering removes test overlaps and pairs with high character 4-gram TF-IDF similarity to test sentences.Source: §F.2 · pp. 17, 19

F.3 FINE-TUNING PARAMETERS

  • Fine-tuning uses full-parameter training for one epoch with InfoNCE, and query-side negatives appear in 50% of examples.Fake-negative masking is omitted because it could mask the query-side distractors the model is intended to learn from.Source: §F.3 · p. 19

F.4 FINE-TUNING INSTRUCTIONS

  • SQuAD QA/STS and Tatoeba bitext/STS use task-specific prompts, which are also used in the main evaluation.The prompts respectively target answer passages, equivalent questions, translations, and semantically similar sentences.Source: §F.4 · p. 19

G LANGUAGE-SPECIFIC RESULTS FOR TATOEBA

The Tatoeba appendix shows language-specific fine-tuning checkpoints and tests whether gains require query-side negatives; without those negatives, distractor-setting performance does not improve.

G LANGUAGE-SPECIFIC RESULTS FOR TATOEBA

H FINE-TUNING WITHOUT QUERY-SIDE NEGATIVES

  • Without query-side negatives, continued SQuAD and Tatoeba fine-tuning does not improve performance in the paraphrase-distractor setting.The comparison uses the same data and training setup, indicating the gains are not from task-specific fine-tuning alone.Source: H FINE-TUNING WITHOUT QUERY-SIDE… · p. 19

I FULL MTEB RESULTS FOR FINE-TUNED MODELS

The appendix reports fine-tuned model benchmark score differences and computational estimates, while noting that fine-tuning without query-side negatives does not improve paraphrase-distractor performance.

I FULL MTEB RESULTS FOR FINE-TUNED MODELS

  • Figure 12 reports fine-tuned-minus-base score differences across 41 MTEB English v2 tasks in seven categories; positive values indicate improvement.The MTEB mean is the arithmetic mean across all tasks.Source: I FULL MTEB RESULTS FOR FINE-TUNED… · p. 19, Figure 12 · p. 21
  • Continued SQuAD and Tatoeba fine-tuning without inserted distractors does not improve paraphrase-distractor performance.Source: Figure 11 · p. 20

J COMPUTATIONAL DETAILS

Loading 2610.10508v1…