PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural Hard-Negative Mining from Positives for Instruction-Following Retrieval

Chaolang Li, Jingyu Li, Zhaocheng Du, Xuyang Xie, Gang Wang, Pengwen Dai

arXiv:2610.36900v1cs.AI

TL;DR

Instruction-following retrieval needs contrastive supervision that makes relevance change with instance-specific instructions, but such pairs are scarce. NHNM mines convertible negatives from existing candidates and reuses original positives as instruction hard negatives, with an automated pipeline that filters the resulting supervision. Its 6,716-instance NHNM-MARCO dataset improves encoder backbones from 568M to 4B parameters across InstructIR and FollowIR.

  • Problem

    Instruction-conditioned contrastive pairs are scarce, while conventional fixed-label tuples let models ignore instance-specific instructions.

    Source: §1 · p. 1

  • Method

    NHNM promotes convertible negatives under compatible instructions and repurposes original positives as instruction hard negatives through an automated cleaning, reversal-construction, and geometry-filtering pipeline.

    Source: §1 · p. 2

  • Results

    NHNM-MARCO’s 6,716 instances improve encoder backbones from 568M to 4B parameters across InstructIR and FollowIR.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The approach creates instruction-conditioned ordering contrasts from corpus-grounded documents without synthesizing long negative passages.

    Source: §6 · p. 8

    Substantially larger training corpora and backbones were not evaluated because of computational cost.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Instruction-following information retrieval (IFIR) requires retrievers to rank documents by both a query and its instance-specific instructions, yet instruction-conditioned contrastive supervision remains scarce. Existing synthetic methods typically fix a positive passage and generate additional negatives, overlooking the hard-negative potential of the original positive. We propose Natural Hard-Negative Mining (NHNM), which promotes convertible negatives under query-compatible instructions and repurposes original positives as natural instruction hard negatives. Its automated Candidate Cleaning, Reversal Construction, and Geometry Filtering pipeline produces compact, high-contrast supervision aligned with the pretrained retrieval space. Applied to MSMARCO, NHNM yields NHNM-MARCO, an IFIR training set of nearly 7K instances. On InstructIR and FollowIR, models with 568M to 4B parameters fine-tuned on NHNM-MARCO achieve strong instruction-following performance and match or surpass substantially larger retrievers in several settings. Controlled analyses and ablations attribute these gains to natural hard-negative supervision rather than model scale or training-data volume.

1 Introduction

Instruction-following retrieval needs supervision that makes relevance change with instance-specific instructions, but such contrastive pairs are scarce. NHNM mines instruction-conditioned reversals from existing candidates, producing a compact dataset that improves encoder backbones and performs strongly against larger retrievers in several settings.

1 Introduction

  • IFIR ranks documents by both the query and instance-specific instructions about aspects, scope, evidence, or exclusions.The paper situates IFIR in personalized search, task-oriented information seeking, and retrieval-augmented generation.Source: §1 · p. 1
  • Instruction-conditioned contrastive pairs are scarce and costly to annotate, while adding instructions to fixed-label tuples lets models ignore them.Effective supervision must expose relevance changes conditioned on the instruction.Source: §1 · p. 1
  • Prior synthetic approaches keep a designated positive fixed and generate negatives around it, overlooking the original positive’s hard-negative potential.Long passage synthesis also increases exposure to generation bias and hallucination.Source: §1 · p. 1
  • Its automated pipeline cleans candidates, constructs reversals, and filters for instruction-induced shifts aligned with the pretrained retrieval space.The pipeline produces high-contrast supervision without synthesizing long negative passages.Source: §1 · p. 2
  • Applying NHNM to MS MARCO yields the 6,716-instance NHNM-MARCO dataset, which improves Qwen3-Embedding-0.6B, Qwen3-Embedding-4B, and BGE-M3; the 0.6B model is strong against much larger retrievers on InstructIR and competitive on FollowIR.Source: §1 · p. 2
  • NHNM promotes convertible negatives under compatible instructions and reuses original positives as instruction hard negatives.The original positive remains query-relevant while violating the added criterion.Source: §1 · pp. 1–2

2 Related Work

Instruction-aware retrieval methods and benchmarks motivate supervision that changes document relevance with instance-level instructions. NHNM reverses the usual positive-centered synthesis direction by promoting an existing candidate under a compatible instruction.

2 Related Work

  • Earlier instruction-aware methods prepend task-level instructions or jointly learn generative and embedding capabilities, but task-level descriptions need not change an individual query’s ranking.InstructIR and FollowIR instead evaluate instance-level instruction sensitivity.Source: §2 · p. 2
  • InstructIR and FollowIR test instance-level instruction sensitivity, showing that strong conventional relevance does not imply rankings change as required.InstructIR uses user-specific intents; FollowIR uses detailed TREC narratives and controlled instruction perturbations.Source: §2 · p. 2
  • Promptriever and InF-IR generate negatives around a fixed positive, leaving role reversals among existing retrieval candidates unexplored.Their approaches require synthesizing long passages that remain query-relevant while violating instructions.Source: §2 · p. 2
  • NHNM promotes an existing convertible negative with a compatible instruction, making the original positive a topically relevant but instruction-violating hard negative.The instruction, rather than topical similarity alone, determines the preferred ranking.Source: §2 · p. 2

3 Preliminaries

IFIR ranks candidate documents using both an instruction and a query, scoring their relevance with a dual-encoder retriever. The paper frames instruction-conditioned mining as finding negatives that can become preferred under a suitable instruction.

3 Preliminaries

  • IFIR ranks candidate documents according to both the information need in the query and the relevance criteria in the instruction.The candidate list contains an instruction-conditioned positive and negative documents.Source: §3 · p. 2
  • A dual-encoder scores documents using cosine similarity between the encoding of the concatenated instruction and query and the document encoding.The paper optimizes the standard InfoNCE objective.Source: §3 · p. 3
  • A negative is convertible when a query-compatible instruction makes it satisfy the intended criterion better than the original positive.Under that instruction, the original positive remains topically relevant but violates the added criterion, becoming a natural instruction hard negative.Source: §3 · p. 3

4 Method and Data Curation Pipeline

NHNM turns query-compatible retrieved negatives into instruction-conditioned positives and repurposes original positives as natural hard negatives. Its three-stage curation pipeline selects semantically valid reversals that fit the pretrained retrieval space.

4 Method and Data Curation Pipeline

  • The pipeline retrieves top-M BM25 documents excluding ground-truth positives, creating a query-related pool of candidates that may become relevant under compatible instructions.These candidates may be partially relevant, address alternative aspects, or provide complementary information.Source: §4 · p. 3

4.1 Stage I: Candidate Cleaning

  • Hybrid rule–LLM cleaning removes malformed, truncated, and near-duplicate passages, then classifies negatives as false, convertible, or other and retains valid positives with at least one feasible convertible negative.A convertible negative must not satisfy the query but must be promotable under a compatible refinement; its proposed instruction preserves query intent and answer type while keeping the original positive relevant but noncompliant.Source: §4.1 · pp. 3–4

4.2 Stage II: Reversal Construction

  • Reversal construction selects the clearest contrast and refines its instruction to the minimum sufficient condition, rejecting intent shifts, answer leakage, and instructions that leave the original positive compliant.The selected convertible negative becomes the instruction-conditioned positive, while the original query-positive passage becomes a natural instruction hard negative without synthesizing a negative passage.Source: §4.2 · p. 4

4.3 Stage III: Geometry Filtering

  • Geometry filtering uses the frozen pretrained encoder to retain instruction-induced shifts in the intended direction, with γ = 0 in the main experiments.The criterion does not require the model to rank the new positive above the original positive before fine-tuning, preserving unsolved reversals as learning signal.Source: §4.3 · p. 4

4.4 NHNM-MARCO Construction

  • The pipeline produces 6,716 NHNM-MARCO instances from MS MARCO-W after rule cleaning, LLM verification, reversal construction, and geometry filtering.It begins with 36,306 tuples; 7,527 valid seeds contain 11,254 convertible-negative candidates, and 7,470 seeds pass reversal construction.Source: §4.4 · p. 4
  • 88% of 100 randomly sampled final instances pass manual checks for intent preservation, valid reversal, natural-hard-negative status, and absence of unsupported constraints or answer leakage.NHNM-MARCO is used to fine-tune Qwen3-Embedding-0.6B and 4B, with transfer also evaluated on BGE-M3.Source: §4.4 · p. 4

5 Experiments

Across InstructIR and FollowIR, NHNM-trained retrievers deliver strong results, with gains across encoder families and a compact 6,716-instance dataset. Controlled comparisons and ablations favor NHNM’s supervision paradigm and identify cleaning, geometry filtering, and the original positive as consequential components.

5.1 Experimental Setup

  • Experiments fine-tune Qwen3-Embedding-0.6B and 4B and BGE-M3, then evaluate on InstructIR and FollowIR with benchmark-specific relevance and instruction-sensitivity metrics.The 0.6B model is fully fine-tuned for one epoch; the 4B and BGE-M3 models use rank-8 LoRA.Source: §5.1 · p. 5

5.2 Baseline Models

  • The baseline comparisons span cross-encoder rankers, bi-encoders, commercial embedding APIs, instruction-following LMs, and fine-tuned retrieval models.The controlled Promptriever-style and InF-IR-style comparisons use the same Qwen3-Embedding-0.6B backbone.Source: §5.2 · p. 5

5.3 Main Results

  • 3.96 Robustness@10 points and 1.02 nDCG@10 points over Promptriever on InstructIR; FollowIR’s 23.7 average relevance score leads fine-tuned retrievers by 2.8 points.NHNM is 1.1 points below FollowIR-7B in average relevance while using a 0.6B dual-encoder backbone.Source: Table 1 · p. 5, §5.3 · p. 5, Table 2 · p. 6

5.4 Controlled Training-Data Paradigm Comparison

  • With backbone, instance count, and training setup fixed, NHNM-MARCO is strongest on both metrics; InF-IR-style training degrades robustness and average ranking quality.Promptriever-style training provides a modest improvement over the unfine-tuned baseline.Source: Table 3 · p. 6, §5.4 · pp. 6–7

5.5 Ablation Study

  • Removing candidate cleaning, geometry filtering, or the original positive weakens instruction-following performance across the reported benchmarks.Cleaning removes noisy candidates; geometry filtering favors shifts compatible with pretrained retrieval space; removing the original positive particularly reduces instruction sensitivity.Source: Table 4 · p. 7, §5.5 · p. 7

5.7 Natural Hard-Negative Geometry Analysis

  • NHNM increases query-negative similarity while preserving comparable positive-negative similarity, and its contrastive relation directions have fewer near-duplicates.Figure 3 compares similarity geometry and the empirical distribution of nearest-neighbor absolute cosine between normalized positive-negative relation directions.Source: Figure 3 · p. 7, §5.7 · p. 8

5.8 Training Data Size Sensitivity

  • At 25% training data both benchmarks beat the base model; performance rises consistently through 75% across InstructIR metrics, FollowIR collection scores, and p-MRR.The analysis uses nested subsets of NHNM-MARCO and fine-tunes for one epoch; zero percent denotes the base model.Source: Figure 4 · p. 7, §5.8 · p. 8

5.6 Training Data Composition Analysis

  • NHNM-MARCO is best across all reported InstructIR metrics; ordinary MS MARCO triples hurt, while positive-based instruction data improves nDCG@10 and MAP but not Robustness@10.The proposed data promotes a naturally retrieved convertible negative and uses the original positive as an instruction-violating hard negative.Source: Table 5 · p. 8, §5.6 · p. 8

6 Conclusion

NHNM converts existing query-positive-negative tuples into instruction-conditioned ordering contrasts by promoting a convertible negative and reusing the original positive as a relevant, instruction-violating hard negative. On MS MARCO, it yields 6,716 training instances and improves encoder backbones across InstructIR and FollowIR; controlled comparisons and ablations attribute the gains to the data paradigm and the original positive.

6 Conclusion

  • NHNM promotes a convertible negative under a concise instruction and reuses the original positive as a topically relevant, instruction-violating hard negative, avoiding passage synthesis and making the instruction necessary for ranking.It turns existing query-positive-negative tuples into instruction-conditioned ordering contrasts.Source: §6 · p. 8
  • Across InstructIR and FollowIR, NHNM-MARCO improves encoder backbones from 568M to 4B parameters, and controlled comparisons and ablations attribute gains to the data paradigm and original positive.Applied to MS MARCO, the pipeline produces 6,716 training instances.Source: §6 · p. 8

Limitations

Computational cost limited evaluation of substantially larger training corpora and backbones; the authors leave large-scale data construction and evaluation to future work.

Limitations

  • Computational cost prevented evaluation of substantially larger training corpora or backbones, leaving large-scale construction and evaluation for future work.Source: Limitations · p. 8

A Training Details

Training details vary by backbone, including geometry filtering and adaptation; runs use a single RTX 3090 with gradient checkpointing and gradient clipping.

A Training Details

  • From the shared 7,470-instance pool, frozen Qwen3-Embedding-0.6B filtering retains 6,716 instances, while 4B filtering retains 6,655.The 0.6B model is fully fine-tuned and the 4B model uses rank-8 LoRA; both results use one-epoch checkpoints.Source: A Training Details · p. 11
  • BGE-M3 recomputes filtering with its original frozen checkpoint, retaining 7,354 instances at γ = 0; its LoRA adapter is merged before dense-only evaluation.The checkpoint reported is after one epoch.Source: A Training Details · p. 11
  • All runs use one RTX 3090 with gradient checkpointing and maximum gradient norm 1.0.Source: A Training Details · p. 11

B More Experiments on FollowIR

The appendix reports FollowIR results for a larger backbone, conventional retrieval retention, token usage, and the prompt stages used to construct the data. It details how candidates are cleaned and how a selected negative is promoted under a rewritten instruction.

B More Experiments on FollowIR

C Conventional Retrieval Retention

D Token Usage

  • Candidate Cleaning accounts for most of the 64.98M API tokens; Reversal Construction refines draft instructions, while rule-based cleaning and Geometry Filtering run locally without API-token cost.Source: D Token Usage · p. 11, Table 9 · p. 13

E Prompts Used in Data Construction

E.1 Stage I: Candidate Cleaning and CN Feasibility

  • Candidate Cleaning validates the positive, classifies every negative as FN/CN/ON, and retains a CN only when its draft instruction preserves query intent while making it better than P.Source: §E.1 · pp. 11–12

E.2 Stage II: Negative-to-Positive Reversal

  • Reversal Construction selects the clearest CN and rewrites its draft as a concise instruction that promotes it while making P fail the added condition.The instruction must preserve the query’s main subject, intent, relation, and answer type; the stage rejects cases without a natural instruction or with an empty negative list.Source: §E.2 · pp. 12–13

F Data-Construction Cases

The construction cases show how cleaning rejects negatives that already answer the query and how an added instruction can make a compatible convertible negative positive while turning the original, still-topical positive into a hard negative.

F Data-Construction Cases

  • Across the cases, cleaning removes false negatives that already answer the query, while geometry-filtered pairs use an added condition to promote a convertible negative and demote the still-topical original positive.The retained negative supplies a compatible discriminating facet rather than mere topic overlap.Source: F Data-Construction Cases · p. 13, Table 10 · p. 14

1. Healing time for broken ribs

  • For broken ribs, the removed false negative already answers the usual recovery-duration question, whereas the convertible negative adds delayed healing absent from the ordinary answer.The final instruction asks about a fracture unhealed after about two months, including non-union assessment and surgery.Source: §1 · p. 14

2. Purpose of the Equal Rights Amendment

  • For the ERA, the removed false negative directly states its purpose, while a compatible convertible negative adds its downstream legal and social impact.Source: §2 · p. 14

3. Causes of blood in urine

  • The blood-in-urine example is marked as a final role reversal; its original positive lists infections, kidney stones, medications, and urinary-tract damage as possible causes.Source: §3 · p. 14

1. Persistent rib fracture

  • For persistent rib fractures, the new positive covers non-union after two to three months and possible surgery, while the original positive lacks those details.Source: §1 · p. 14

2. ERA purpose and downstream impact

  • In the ERA case, the new positive connects gender equality with later legal and social change, while the original positive states the purpose but not its impact.Source: §2 · p. 14

3. Critical glucose levels in newborns

  • For newborn glucose levels, the new positive distinguishes healthy full-term from premature babies, unlike the original positive’s general threshold.Source: §3 · p. 14

4. An outline as a prewriting framework

  • The outline example keeps the same entity and topic while making relevance depend on the instruction-induced boundary, rather than on coarse topical dissimilarity.Source: §4 · p. 14

G Case Studies

Case studies on InstructIR and FollowIR illustrate instruction-specific ranking: the fine-tuned model favors passages satisfying added criteria over topically related but mismatched results. The authors also identify ambiguity or bias in generated instructions as a risk and describe mitigation steps.

G Case Studies

  • In the InstructIR case, the one-epoch NHNM-MARCO model ranks the annotated drug-interaction passage first, unlike the baseline’s topically related but instruction-mismatched results.The query asks for a definition, while the instruction narrows the need to evidence about interactions with cardiovascular and metabolic drugs.Source: G Case Studies · pp. 14–15, Table 11 · p. 15
  • In the FollowIR case, the model removes a leukemia-focused document that violates the exclusion while retaining leading results about mobile-phone exposure and brain tumors.The baseline’s fifth result is topically close but non-relevant under the modified criterion.Source: G Case Studies · p. 15, Table 12 · p. 16

H Ethical considerations

  • LLM-generated instructions may inherit ambiguity or bias; the authors mitigate this with rule-based checks, joint semantic verification, and geometry filtering, and do not use generated evaluation labels.Source: H Ethical considerations · p. 16
  • A generative AI assistant supported manuscript formatting and wording, while GPT-5.1 is also used in the data-construction method.The authors say they reviewed the resulting text and formatting and remain responsible for the paper’s claims, citations, and presentation.Source: H Ethical considerations · p. 16

Related papers

Loading 2610.36900v1…