PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-improving data synthesis can scale embedding models across domains and tasks

Zucan Lyu, Rui Tang, Jianlyu Chen, Peisheng Jin, Zheng Liu, Defu Lian

arXiv:2610.14730v1cs.IRcs.AIcs.LG

TL;DR

Adapting embedding models requires training data that reflect task-specific needs and relevance criteria, while manual data design is costly to scale. SYNTRA refines reusable synthesis instructions using few-shot examples and feedback on trial outputs. Its generated data improved retrieval across tasks and model settings, and performance continued to rise as training data scaled on MS MARCO.

  • Problem

    Adapting retrieval models requires training data reflecting task-specific information needs and relevance criteria, but manual data design is costly and difficult to scale.

    Source: §2 · p. 12

  • Method

    SYNTRA uses few-shot query–positive-document examples and feedback on trial outputs to refine shared synthesis instructions for reusable data generation.

    Source: §2 · p. 12

  • Results

    Automatic self-improvement was competitive with human calibration on most tested tasks, SYNTRA data benefited strong embedding models, and MS MARCO retrieval gains continued as training data scaled.

    Source: §2 · p. 13

  • Takeaways & Limitations

    Refined instructions provide reusable guidance for constructing task-specific supervision, with production generation using a locally deployed model after small-scale calibration.

    Source: §2 · pp. 12–13

Abstract

from arXiv · show

Adapting embedding models to new tasks requires training data that reflect task-specific information needs and relevance criteria. However, manual data design is costly and difficult to scale. Here we introduce SYNTRA, a novel retrieval data synthesis method whose self-improvement loop iteratively refines reusable synthesis instructions using feedback on trial outputs. This process requires only an unlabelled corpus and a few relevant query-document pairs. Experiments across 24 retrieval datasets spanning 15 domains demonstrate the effectiveness of the synthesized data. These data also improve widely used, strong embedding models on tasks with and without existing labelled training sets. Self-improvement enhances retrieval on all six evaluated tasks, outperforming human calibration on four. Moreover, retrieval performance continues to improve on MS MARCO as we scale the training dataset to 32 times its initial size. Overall, with minimal human intervention, SYNTRA enables scalable adaptation of embedding models across domains and tasks.

1 Results

Across 24 retrieval datasets in 15 domains, SYNTRA-generated data improved retrieval over the MS MARCO-trained baseline in every task-specific comparison. The results also show gains from self-improvement, synthetic queries, and scaling, with benefits for strong embedding models and multiple synthesis components.

1.1 One synthesis workflow improves embeddings across domains and tasks

  • SYNTRA-generated data improved retrieval over the MS MARCO-trained baseline in all 23 task-specific comparisons across 24 datasets and 15 domains.The comparisons span legal, financial, scientific, biomedical, medical, coding, mathematical, debate, entity retrieval, spatial reasoning and commonsense reasoning.Source: §1.1 · pp. 5–6, Figure 4 · p. 7

1.2 Self-improvement enhances the training value of synthetic data

  • Automatic self-improvement increased training value across all six tasks; on NFCorpus and FiQA, refined-instruction data surpassed the MS MARCO baseline after initial-instruction data fell below it.Automatic self-improvement scored higher than human calibration on four of six tasks; human calibration scored higher on TheoremQA theorems, while Text2SQL scores were close (62.240 versus 62.449). In the FiQA case study, feedback clarified that an answer correcting a query’s premise could be relevant.Source: Figure 4 · p. 7, §1.2 · p. 8

1.3 Synthetic queries outperform human-generated queries in retrieval training

  • Under matched training conditions, SYNTRA-generated queries produced higher mean nDCG@10 on BEIR and NanoBEIR than human-generated queries.The models trained with SYNTRA-generated queries scored 42.87 versus 42.18 on BEIR and 55.51 versus 54.53 on NanoBEIR; they also achieved competitive performance against task-specific query sources across four tasks, scoring higher on TheoremQA questions and CosQA.Source: §1.3 · pp. 8–10

1.4 Generated labels agree with human relevance judgements

  • LLM relevance labels achieved 91.3% binary agreement and 68.7% three-class agreement with human references; pooled precision for positive and hard-negative training labels was 82.8%.Most three-class disagreements concerned hard versus easy negatives.Source: §1.4 · p. 11

1.5 Gains extend to widely used, strong embedding models

  • Adding SYNTRA-generated data improved BGE-M3 and Qwen3-Embedding-8B on all three tested tasks with existing labelled training sets; both models also improved on all three tasks without such sets.For tasks with labels, gains were measured against fine-tuning on the existing task-specific sets alone; for tasks without labels, they were measured against the corresponding pretrained models without task-specific fine-tuning.Source: §1.5 · p. 11

1.6 Scaling synthetic data yields continued retrieval gains

  • MS MARCO nDCG@10 improved at each of six evaluated data scales, rising from 30.88 to 33.20 as SYNTRA-generated training data increased.The evaluation used a fixed training recipe and protocol, scaling from 10,000 to 320,000 examples.Source: Figure 6 · p. 10, §1.6 · p. 12

1.7 Core design choices improve retrieval performance

  • On NanoBEIR, removing semantic identifiers and query attributes, task-adapted query instructions, or relevance annotation reduced mean nDCG@10 from the full workflow’s 54.64.Replacing relevance annotation with hard-negative mining scored 52.80; adding source-document relevance annotation scored 53.75. Generating more queries per document also improved both evaluated tasks: CodeTrans-DL rose from 37.048 to 38.374 and MedicalQA from 33.813 to 39.721 as query count increased from one to four.Source: Figure 6 · p. 10, §1.7 · p. 12

2 Discussion

SYNTRA adapts reusable synthesis instructions to task-specific retrieval needs through few-shot-guided feedback, then reuses them for large-scale data production. The authors report benefits across the evaluated retrieval settings and propose extending self-improvement to other training-data synthesis tasks.

2 Discussion

  • Few-shot examples and feedback on trial outputs adapt shared synthesis instructions, which are reused to produce training data at scale.The authors say commercial-model calls are concentrated in small-scale calibration, supporting economical synthesis.Source: §2 · pp. 12–13
  • Refined instructions exceeded the MS MARCO-trained baseline on NFCorpus and FiQA, while data from initial instructions fell below it.Automatic self-improvement was competitive with human calibration on most tested tasks, though human calibration remained advantageous on TheoremQA (theorems).Source: §2 · p. 13
  • SYNTRA-generated data benefited strong embedding models, supplying supervision where labelled training sets were unavailable and complementing existing supervision elsewhere.On MS MARCO, retrieval gains continued as the training data scaled up across the evaluated scales.Source: §2 · p. 13
  • The authors propose extending self-improvement to other LLM capabilities, with feedback assessing whether generated examples meet task requirements.They also suggest using evaluation, execution and user-interaction feedback to identify unmet training needs.Source: §2 · p. 13

3 Methods

The methods prepare a corpus and task definition, refine shared synthesis instructions using trial outputs, and reuse those instructions to create retrieval training data. The paper also describes model evaluation, human assessment of relevance labels, and NanoBEIR’s use as a lower-cost screening proxy.

3.1 Corpus preprocessing and task preparation

  • Corpus preparation concatenates available titles and bodies, encodes documents as normalized vectors, and selects diverse source documents while retaining a full-corpus retrieval index.Task preparation infers query type, document type and relevance criteria from few-shot examples, with the option for users to provide a definition directly.Source: §3.1 · pp. 13–14

3.2 Task adaptation through self-improvement

  • Test-mode generate–evaluate–refine cycles use feedback against task requirements and examples to revise shared instructions before production.Identifier and attribute guidance, query instructions, and relevance criteria are refined in distinct loops; the latter checks supplied positive pairs and infers negative criteria from task requirements.Source: §3.2 · pp. 14–15

3.3.1 Identifier extraction and attribute construction

  • Production extracts document-grounded identifiers and task-appropriate query attributes, then combines them to plan distinct information needs.A query-writing stage expresses each plan as a query using the source document and task-adapted instructions.Source: §3.3.1 · p. 16, §3.3.2 · p. 16

3.3.3 Relevance-label synthesis

  • Generated queries retrieve top-K candidates from the full corpus, which are independently judged on a four-point relevance scale and mapped to positives, hard negatives and easy negatives.Training records retain queries with at least one positive and usable negative; exports can use hard negatives alone or supplement them with easy negatives.Source: §3.3.3 · p. 16, Training records · p. 17

Embedding models.

  • The main cross-task evaluation trains task-specific Qwen3-0.6B models with LoRA and compares them with an MS MARCO-trained baseline; other experiments test synthetic data with and without existing labelled sets.Query-source comparisons use shared retrieval, annotation and training procedures within tasks, and the data-scaling experiment holds the training recipe and evaluation protocol fixed across six data sizes.Source: Embedding models · p. 17, Query-source comparisons · p. 17, Data-scaling experiment · p. 18

3.5 Human evaluation of relevance labels

  • Human reference labels came from seven annotators, and agreement was measured using exact matches and Cohen’s κ, with Fleiss’ κ used for inter-annotator agreement.NanoBEIR evaluation averages nDCG@10 across 13 tasks, with each task contributing equally to the suite score.Source: §3.5 · pp. 18–19, §3.6 · p. 19

3.6 Consistency between NanoBEIR and BEIR

  • NanoBEIR closely preserved BEIR model ordering within encoder-only models (ρ = 0.933) and decoder-only models (ρ = 0.817), with median absolute rank changes of one position in both groups.The authors support NanoBEIR for lower-cost screening, ablations and hyperparameter selection, while cautioning that absolute nDCG@10 scores are not interchangeable and local rank reversals remain.Source: §3.6 · pp. 19–20

Competing interests.

The authors report no competing interests and identify supplementary materials and experimental settings.

Competing interests.

Supplementary information

  • Supplementary information includes workflow prompts, retrieval-task definitions, initial and refined instructions for a FiQA case study, and key experimental settings.Source: Competing interests · pp. 25–26

Extended Data

The Extended Data passages describe production-time reporting, relevance-label agreement measures, and the scope of data-size and retrieval-performance tables.

Extended Data Table 1

  • Extended Data Table 1 reports production-stage runtimes and finalised training-data sizes for selected batches, excluding calibration, indexing, and model training and evaluation.The caption cautions that these are selected batches, not task-average runtimes or total experimental costs.Source: Extended Data · p. 27

Extended Data Table 2

  • Extended Data Table 2 compares LLM-generated labels with human references using agreement, Cohen’s κ, and class-averaged precision, recall and F1.Its caption distinguishes three-class evaluation from binary evaluation, which merges the two negative categories.Source: Extended Data · p. 28

Extended Data Table 3

  • Extended Data Table 3 reports confusion-matrix and class-wise label results; pooled precision is 82.8% among the 87 LLM-labelled positives or hard negatives.The caption identifies these as the categories retained in the hard-negative-only training export.Source: Extended Data · p. 28

Extended Data Table 4

  • Extended Data Table 4 pairs query and training-data sizes with nDCG@10 scores for task-specific query-source comparisons against a shared MS MARCO-trained baseline.The caption notes that the two positive-selection variants are concatenated without deduplication, so combined input rows are twice the per-variant count.Source: Extended Data · p. 29

Supplementary Information

The Supplementary Information collects the synthesis prompts, task definitions, FiQA instructions, and experimental configurations. Its contents are organized as a prompt-and-instructions note and experimental-settings materials.

Supplementary Information

  • The supplementary materials include complete prompts for the principal synthesis workflow, retrieval-task definitions, FiQA instructions, and experimental configurations.Shared prompt text is displayed once; experimental settings are summarized in Supplementary Note 2 and Tables 1 and 2.Source: Supplementary Information · p. 31

Supplementary Note 1: Prompts and task-specific instructions S2

  • Supplementary Note 1 lists source conventions and task-definition, identifier/attribute, query-generation, and relevance-annotation materials.It also includes optional code branches and task descriptors with registered examples.Source: Supplementary Note 1: Prompts and… · p. 31
  • Production relevance annotation instructions are separately listed among the supplementary prompt materials.The contents list places this item at S17.Source: Supplementary Note 1: Prompts and… · p. 31
  • The FiQA case study’s complete initial and refined instructions appear in the supplementary materials.The contents list places them at S21.Source: Supplementary Note 1: Prompts and… · p. 31

Supplementary Note 1: Prompts and task-specific instructions

The appendix documents prompt inputs and formatting, then gives prompts for inferring, validating, and refining task definitions. Together, these materials specify how few-shot examples shape reusable task requirements.

Source and input conventions

  • Input conventions distinguish runtime fields from shared display blocks, which must be expanded before prompts are sent.The appendix also describes JSON serialization, example formatting, and trial/candidate inputs, including sample limits and identifier mapping.Source: Source and input conventions · p. 32
  • The appendix names task-specific fields for task descriptions, documents, query instructions, annotations, feedback, and trial samples.The task-definition caller uses up to five examples from each reference task, omitting references identical to the target task.Source: Source and input conventions · p. 32

Prompt 1: Task-definition generation

  • Task-definition generation infers query type, document type, and relevance criteria from labelled examples, using contrastive analysis when hard negatives are available.Without hard negatives, the prompt focuses on the aspects that make positive documents ideal matches.Source: Prompt 1: Task-definition generation · p. 33

Prompt 2: Task-definition validation

  • The task-definition validation prompt checks whether inferred query, document, and relevance descriptions accurately characterize the examples and distinguish the target task.Its criteria also assess whether the definition is actionable for query generation and document retrieval.Source: Prompt 2: Task-definition validation · p. 34
  • The validation output marks a definition production-ready when it passes with an overall score of at least 3.The rubric calls for specific feedback and actionable revisions when examples reveal systematic mismatches.Source: Prompt 2: Task-definition validation · p. 34

Prompt 3: Task-definition refinement

  • Task-definition refinement rewrites the definition to address evaluation feedback while grounding every claim in examples and preserving the prescribed template.The output specifies query and document types as 2–5 word phrases and relevance as one precise sentence.Source: Prompt 3: Task-definition refinement · pp. 34–35

Identifier and attribute guidance

The identifier/attribute workflow generates document-grounded information anchors and task-aligned query-diversity dimensions, then evaluates and refines their reusable guidance. Its prompts emphasize diversity that remains useful for the target retrieval task.

Prompt 4: Identifier/attribute generation

  • Identifier and attribute generation produces document-grounded anchors and independent dimensions for varying query expression, guided by task context and relevance criteria.The prompt requires 3–10 identifiers and 3–6 attributes, each with 2–4 mutually exclusive, actionable options.Source: Prompt 4: Identifier/attribute… · pp. 35–36

Prompt 5: Identifier/attribute validation

  • Validation checks whether identifiers are diverse, non-redundant, and grounded in documents, whether attributes align with the query task and offer valid mutually exclusive options, and whether both support concrete, natural queries.Source: Prompt 5: Identifier/attribute… · p. 37

Prompt 6: Identifier/attribute refinement

  • Refinement converts evaluation feedback into concise, reusable task-level guidance for future identifier and attribute generation.It abstracts recurring issues into general principles, preserves what works, and specifies actionable guidance in 100–300 words.Source: Prompt 6: Identifier/attribute… · pp. 38–39

Query-generation instructions

The query-instruction prompts derive reusable synthesis guidance from few-shot examples, validate generated queries against task and style requirements, and revise instructions using feedback. They emphasize authentic query-producer perspective as well as relevance and style.

Prompt 7: Instruction construction

  • Instruction construction derives a synthesizer role and query-extraction steps from few-shot query–positive-document pairs.The steps should generalize across documents and guide query framing from the producer’s perspective without copying example-specific content.Source: Prompt 7: Instruction construction · pp. 39–40

Prompt 8: Style guidance

  • Style guidance distills linguistic traits, relevance heuristics, and anti-patterns from task context and few-shot queries.The resulting guide is intended for use as a system prompt for query generation.Source: Prompt 8: Style guidance · p. 40

Prompt 9: Instruction validation

  • The validation process evaluates pairs individually, summarizes task-level issues, and returns feedback for refining reusable instructions.The prompt asks that feedback generalize underlying issues rather than preserve sample-specific details.Source: Prompt 9: Instruction validation · p. 41
  • Query validation compares generated pairs with golden queries for style, task alignment, naturalness, and authentic producer intent.It also evaluates whether the instruction’s role and extraction process effectively guide generation.Source: Prompt 9: Instruction validation · p. 41
  • The rubric treats queries as production-ready when they closely match examples in style, authenticity, naturalness, and query-document relationship.Systematic authenticity or style mismatches require a failing verdict.Source: Prompt 9: Instruction validation · pp. 41–42

Prompt 10: Instruction refinement

  • Instruction refinement rewrites query-synthesis guidance to address feedback while preserving effective elements and sharpening role or extraction steps when needed.The revised instruction is formatted as compact, embeddable guidelines.Source: Prompt 10: Instruction refinement · p. 42

Query synthesis

Query synthesis separates planning diverse information needs from realizing each plan as a task-specific query. Blueprints select identifiers and attribute options, which query generation must preserve.

Query synthesis

  • Shared inputs provide a document, identifiers, query attributes, task context, examples, style guidance, and the source document as the sole truth.Planning chooses information needs; realization writes one query per supplied blueprint.Source: Query synthesis · pp. 42–43

Prompt 11: Blueprint planning

  • Blueprint planning selects exactly one identifier and one option per attribute dimension for each plan, maximizing coverage while avoiding repeated combinations.Plans select but do not write queries.Source: Prompt 11: Blueprint planning · pp. 43–44

Prompt 12: Query realization from blueprints

  • Query realization writes one distinct query per blueprint, reflecting its assigned attributes and identifier without changing those selections.The output retains the identifier and attributes as separate JSON fields.Source: Prompt 12: Query realization from… · pp. 44–45

Relevance-annotation instructions

The prompts construct task-specific relevance instructions, use calibration mismatches to generate refinement guidance, and rewrite instructions while preserving the annotation contract.

Prompt 13: Annotation-instruction construction

  • The annotation instruction defines task-specific positive, hard-negative, and easy-negative categories using an absolute 0–3 scoring rubric.Scores 3 and 2 map to positive, 1 to hard_negative, and 0 to easy_negative.Source: Prompt 13: Annotation-instruction… · p. 45
  • The instruction emphasizes query-document relevance boundaries, especially distinguishing plausible hard negatives from documents that satisfy the task criteria.It also requires sequential, independent evaluation of each candidate and valid JSON output.Source: Prompt 13: Annotation-instruction… · p. 45

Prompt 14: Calibration-feedback generation

  • Calibration feedback starts from confirmed label mismatches and diagnoses instruction gaps that may have permitted the errors.The calibration labels have already been compared with reference labels.Source: Prompt 14: Calibration-feedback… · p. 46
  • Feedback groups recurring failure patterns and turns them into concrete, task-specific wording changes rather than simply listing errors.The prompts direct the model to use mismatches as evidence for generalizable guidance, including hard-negative boundaries.Source: Prompt 14: Calibration-feedback… · p. 46
  • The feedback prompt supplies task context, current instructions, calibration results, and confirmed misclassifications for diagnosis.It requests a global failure summary, positive and negative issues, instruction gaps, and suggestions.Source: Prompt 14: Calibration-feedback… · p. 46

Prompt 15: Annotation-instruction refinement

  • Instruction refinement rewrites the prompt to address feedback while retaining the 0–3 scoring system and multi-document JSON output contract.It also requires task-specific criteria, clear positive–hard-negative boundaries, and preservation of effective instruction components.Source: Prompt 15: Annotation-instruction… · p. 47

Production relevance annotation

Production annotation applies the refined task-specific instruction to queries and candidate documents, with local identifiers tied to positions in candidate chunks.

Production relevance annotation

Optional branches in the accompanying code

The accompanying code retains additional prompt templates with distinct functions or output contracts.

Optional branches in the accompanying code

Task descriptors and registered examples

The task registry defines retrieval tasks through query types, document types, and explicit relevance criteria, with 121 examples spanning 24 tasks. Its descriptors distinguish task-specific needs, from legal comparability and financial usefulness to functional equivalence, evidence, and reasoning.

Task descriptors and registered examples

  • The registry contains 121 examples across 24 retrieval tasks, usually five per task, and records each task’s query type, document type, and complete relevance definition.It includes six examples for DBpedia-entity.Source: Task descriptors and registered examples · p. 48

LeCaRDv2; FiQA; SCIDOCS; SciRepEval Search

  • LeCaRDv2 requires legal comparability; FiQA requires material help with a financial need; SCIDOCS and SciRepEval Search define relevance through scholarly relationships or direct satisfaction of a literature query.Source: LeCaRDv2 · p. 48, FiQA · p. 48, SCIDOCS · p. 48

CQADupStack-Stats; Natural Questions; CQADupStack-Physics

NFCorpus; EvidenceBench; MedicalQA; MedicalRetrieval

CodeTrans-DL; CosQA; Text2SQL

BIRCO-relic; TheoremQA questions; TheoremQA theorems; CQADupStack-Mathematica

Touche2020; ArguAna; DBpedia-entity

SpartQA; WinoGrande

  • SpartQA evaluates answers against stated spatial relations, including both-or-neither cases; WinoGrande requires a completion consistent with the sentence’s commonsense constraints.Source: SpartQA · p. 50, WinoGrande · p. 51

FiQA case study: complete initial and refined instructions

The FiQA case study contrasts initial and refined synthesis instructions for query generation and relevance annotation. Refinement steers queries toward users’ financial needs and clarifies that correcting a query’s premise can qualify as relevant.

FiQA case study: complete initial and refined instructions

Query-generation example and instructions

Relevance-annotation example and instructions

  • The initial annotation instructions define positive, hard-negative, and easy-negative scores by whether a document materially addresses the query’s financial need.Source: Relevance-annotation example and… · p. 54
  • The annotation protocol requires independent scoring of each candidate and a JSON output listing scores, classifications, reasoning, and document-id groups.Source: Relevance-annotation example and… · p. 54
  • The refined rubric treats a direct resolution—including refuting a mistaken premise—as positive, rather than labeling it irrelevant for failing to support the proposed action.Source: Relevance-annotation example and… · p. 55
  • Hard negatives include topical overlap with the wrong decision context, missing constraints, outdated or inapplicable information, and answers that solve a different problem.The mortgage example illustrates that a direct answer showing a proposed action will not work can still be relevant.Source: Relevance-annotation example and… · p. 55

Supplementary Note 2: Experimental configurations

The supplementary note specifies model configurations, shared training settings, and human-evaluation sampling for experiments on self-improvement, model adaptation, and data scaling.

Supplementary Note 2: Experimental configurations

Related papers

Loading 2610.14730v1…