Source-linked AI summary
Dual-View Training for Instruction-Following Information Retrieval
Qingcheng Zeng, Puxuan Yu, Aman Mehta, Fuheng Zhao, Rajhans Samdani
TL;DR
Instruction-following retrievers often miss explicit relevance constraints because they rely on semantic query-document similarity. The paper synthesizes complementary instructions that reverse document relevance and trains on both views, improving instruction sensitivity while preserving general retrieval quality at matched data budgets.
Problem
Most retrievers fail to internalize detailed instruction-defined relevance criteria and instead rely on superficial query-document similarity.
Method
An LLM generates complementary instructions that reverse the relevance labels of positive and instruction-negative documents, creating dual-view training pairs.
Results
45%: FollowIR p-MRR improves with the 305M-parameter encoder, and matched-budget experiments show dual-view data improves both instruction sensitivity and general retrieval quality.
Takeaways & Limitations
Dedicated instruction data drives instruction sensitivity, while dual-view synthesis reconciles instruction supervision with the data volume needed for general retrieval quality.
Takeaways & Limitations
The approach assumes meaningful complementary instructions exist, and the experiments focus on encoder bi-encoders and English-language benchmarks.
Abstract
from arXiv · showhide
Instruction-following information retrieval (IF-IR) studies retrieval systems that must not only find documents relevant to a query, but also obey explicit user constraints such as required attributes, exclusions, or output preferences. However, most retrievers are trained primarily for semantic relevance and often fail to distinguish documents that match the topic from those that satisfy the instruction. We propose a dual-view data synthesis strategy based on polarity reversal: given a query, a document that is relevant under the instruction, and a hard negative that matches the query but violates the instruction, we prompt an LLM to generate a complementary instruction under which the two documents swap relevance labels. By presenting the same document pair under complementary instructions that invert their relevance labels, the training signal forces the retriever to reconsider the same candidate set through the instruction, rather than relying on fixed topical cues. On a 305M-parameter encoder, our method improves performance on the FollowIR benchmark by 45%, surpassing general-purpose embedding models of comparable or larger scale. Through head-to-head comparisons at matched data budgets, we further show that data diversity and instruction supervision play complementary roles: the former preserves general retrieval quality, while the latter improves instruction sensitivity. These results highlight the value of targeted data synthesis for building retrieval systems that are both broadly capable and instruction-aware.
1 Introduction
Instruction-following retrieval requires systems to satisfy both query semantics and explicit relevance constraints, yet many retrievers remain driven by superficial query-document similarity. The paper addresses this gap with dual-view polarity-reversal synthesis, which makes the same documents relevant under complementary instructions.
- IF-IR requires retrievers to follow explicit constraints about attributes, style, exclusions, or length in addition to matching the query topic.
- Most current models fail to internalize detailed relevance criteria and largely ignore instruction-specific constraints.The cited evaluation measures instruction sensitivity with p-MRR.
- Polarity-reversal synthesis prompts an LLM to generate complementary instructions that swap the relevance labels of an existing positive and instruction-negative pair.
- Dual-view training reuses the same documents under complementary instructions, forcing attention to fine-grained instructional differences rather than surface similarity.
- 45%: the method improves FollowIR p-MRR on a 305M-parameter encoder and surpasses comparable or larger general-purpose embedding models.
2 Methodology
The method creates a complementary instruction for each query-document pair so that the original positive and instruction negative exchange roles. Contrastive training then requires opposite retrieval decisions for the same documents under the two instruction views.
- Given q, D+, D−, and an original instruction, an LLM synthesizes a coherent new instruction under which D− becomes positive and D+ becomes negative.
- Each augmented batch contains both the original and polarity-reversed views, with high similarity assigned to the corresponding positive document in each view.
- The dual objective penalizes instruction-agnostic representations because the two instruction-conditioned queries must retrieve opposite documents.
- The synthesis pipeline uses Qwen3-Next-80B-A3B-Instruct and produces one complementary training sample per seed instance.
3 Experimental Setup
Experiments use two encoder backbones, size-matched original and dual-view training configurations, and a common contrastive training protocol. Evaluation covers instruction sensitivity and overall retrieval quality across multiple benchmark suites.
- The primary encoder is gte-multilingualmlm-base with 305M parameters, while bge-m3-retromae tests cross-backbone generalizability.
- Ins-orig uses 480k original instruction samples, whereas Ins-DV combines 240k original and 240k dual-view synthetic samples.
- All-orig and All-DV each scale to approximately 880k samples while keeping approximately 440k instruct samples fixed.
- Training uses 30 hard negatives per query, InfoNCE loss with τ = 0.02, concatenated query-instruction encoding, and a 512-token maximum sequence length.
- FollowIR reports p-MRR for instruction sensitivity and Score for overall retrieval quality, alongside InfoSearch and MAIR evaluations.
4 Results and Analysis
Dual-view data improves instruction-following metrics, while the best matched-budget configuration combines instruction supervision with sufficient data diversity. The pattern replicates across encoder backbones, with non-instruct supplementation instead degrading instruction sensitivity.
- Instruct-only comparison: 45%: Ins-DV raises FollowIR p-MRR from 5.21 to 7.57 over size-matched Ins-orig, while FollowIR Score falls from 21.33 to 19.73.
- Mixed-data comparison: All-DV raises FollowIR p-MRR from 5.27 to 8.30 and Score from 20.85 to 21.38 over All-orig.
- Mixed-data comparison: On InfoSearch, All-DV achieves positive p-MRR values of 31.91 for length and 12.13 for keyword, whereas All-orig reaches −23.22 and −49.65.
- The role of data diversity: Data diversity sustains general retrieval quality, while instruction supervision drives instruction-following capability; non-instruct supplementation dilutes the instruction signal.
- Cross-backbone generalizability: On bge-m3-retromae, All-DV again achieves the best FollowIR p-MRR at 13.92 while maintaining competitive Score of 20.99.
- Cross-backbone generalizability: The same instruction-sensitivity and data-mixing patterns replicate across both encoder backbones.
5 Discussion
Polarity reversal reframes instruction negatives as conditionally relevant documents under complementary instructions, creating supervision that distinguishes instructional differences rather than mere exclusions. The discussion also links data mixing to a tension between general retrieval quality and instruction sensitivity.
- Polarity reversal versus instruction-based negatives: Polarity reversal treats an instruction negative as relevant under a different, complementary instruction rather than as a permanently failed retrieval.This reframing creates contrastive supervision across instruction space.
- Polarity reversal versus instruction-based negatives: Synthesizing complementary instructions forces the encoder to resolve where instructions diverge, targeting distinctions rather than only exclusions.The resulting signal is described as structurally richer than conventional instruction-negative supervision.
- A gradient perspective on data mixing: At a 50/50 mix, non-instruction data can overwhelm instruction supervision, making instruction sensitivity a fragile property requiring consistent training signal.The discussion contrasts this with competitive instruction-following performance from approximately 38k specialized triplets.
6 Conclusion
The paper presents dual-view data synthesis through polarity reversal, generating complementary training pairs without additional annotation cost. Across two encoder backbones, the approach is reported to improve both instruction-following sensitivity and general retrieval quality at equal data budgets.
- Contribution: The paper proposes dual-view data synthesis based on polarity reversal.The strategy creates complementary training pairs.
- Contribution: Complementary training pairs are created at no additional annotation cost.The synthesis repurposes existing data rather than requiring new annotations.
- Findings: Across two encoder backbones, dedicated instruction data drives IF sensitivity while data diversity sustains general retrieval quality.The result identifies complementary roles for the two data properties.
- Findings: At equal data budgets, dual-view synthesis reconciles the competing demands of instruction sensitivity and general retrieval quality.The approach requires no changes to existing pipelines.
Limitations
The method assumes that each data point admits a meaningful complementary instruction, and its evaluation is limited to encoder-based bi-encoders and English-language benchmarks.
- Scope and assumptions: Meaningful complementary instructions exist for most data points, but narrow relevance criteria may occasionally produce less natural reversals.This limitation is based on manual inspection.
- Scope and assumptions: The experiments evaluate encoder-based bi-encoder retrievers and English-language benchmarks, leaving decoder-based, cross-encoder, and multilingual settings for future work.The stated scope boundaries concern architecture and language coverage.
A The Prompt Template for Data Synthesis
The prompt template supplies a query, original instruction, positive passage, and instruction negatives, then asks for a concise new instruction that reverses one specified relevance judgment. It defines relevance categories, recommends attribute-based reversal levers, and applies diversity and sanity-check guardrails.
- Goal: The goal is to make the original positive negative and the specified negative positive while preserving the status of all remaining negatives.The reversal applies only to the specified passages.
- Output: The requested output is an XML answer containing a new_instruction, with None permitted when the task is judged too difficult.The template explicitly specifies the output wrapper.
- Definitions: The template defines pure-query relevance as satisfying user intent without extra instruction and instruction negatives as query-relevant documents excluded by the instruction.Examples of exclusion include scope, geography, timeframe, format, source, and audience constraints.
- Method: Reversal levers include domain, region, timeframe, audience, style, format, methodology, required artifacts, and explicit exclusions.The method profiles passage attributes before choosing constraints that distinguish the target negative from the positive.
- Guardrails: The new instruction must differ in format and perspective from the original, avoid meta-task language and passage IDs, and not modify the query or passages.The checklist also requires concision of at most two sentences and diversity from the original instruction.
- Inputs: The template takes a query, original instruction, positive passage, specified instruction negative, and remaining negatives as inputs.The remaining instruction negatives may be an empty array.