PaperlayerView exact revision on arXiv

Source-linked AI summary

SearchInstruct: Enhancing Domain Adaptation via Retrieval-Based Instruction Dataset Creation

Iman Barati, Mostafa Amiri, Heshaam Faili

arXiv:2509.10708v1cs.CL

TL;DR

Creating structured, domain-specific SFT data is challenging. SearchInstruct expands seed questions and retrieves relevant documents to ground answers, with reported improvements in specialized-domain performance and factual updates. The paper also identifies limits in deeper knowledge integration and potential retrieval bias.

  • Problem

    Creating structured SFT datasets tailored to specialized domains is challenging, despite their importance for training LLMs.

    Source: §1 · p. 1

  • Method

    SearchInstruct expands domain-specific seed questions with an LLM and retrieves relevant documents to generate context-grounded answers for SFT.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    SearchInstruct improves performance in Iranian cultural domains and supports factual model updates, with measurable gains reported without degrading existing knowledge.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Retrieval-grounded dataset construction can support domain-aligned SFT and factual updates to LLMs in the evaluated scenarios.

    Source: §5 · p. 8

    Surface-level edits may add specific facts without deeper conceptual integration, and web retrieval can propagate source bias into generated data.

    Source: §4.2.1 · p. 8, §6 · p. 9

Abstract

from arXiv · show

Supervised Fine-Tuning (SFT) is essential for training large language models (LLMs), significantly enhancing critical capabilities such as instruction following and in-context learning. Nevertheless, creating suitable training datasets tailored for specific domains remains challenging due to unique domain constraints and data scarcity. In this paper, we propose SearchInstruct, an innovative method explicitly designed to construct high quality instruction datasets for SFT. Our approach begins with a limited set of domain specific, human generated questions, which are systematically expanded using a large language model. Subsequently, domain relevant resources are dynamically retrieved to generate accurate and contextually appropriate answers for each augmented question. Experimental evaluation demonstrates that SearchInstruct enhances both the diversity and quality of SFT datasets, leading to measurable improvements in LLM performance within specialized domains. Additionally, we show that beyond dataset generation, the proposed method can also effectively facilitate tasks such as model editing, enabling efficient updates to existing models. To facilitate reproducibility and community adoption, we provide full implementation details, the complete set of generated instruction response pairs, and the source code in a publicly accessible Git repository: [https://github.com/mostafaamiri/SearchInstruct](https://github.com/mostafaamiri/SearchInstruct)

1 Introduction

SearchInstruct addresses the challenge of creating diverse, realistic, domain-specific SFT data by expanding seed questions with an LLM and retrieving documents to ground answers. The paper reports improved model performance and instruction-following accuracy in specialized domains, as well as effective model editing.

1 Introduction

  • Domain-specific SFT requires structured datasets, but existing approaches can be labor-intensive, lack coverage or task diversity, misrepresent user needs, or rely on outdated internal knowledge.The dataset must reflect the diversity and complexity of real-world queries.Source: §1 · p. 1
  • SearchInstruct expands a small set of seed questions with an LLM, then retrieves relevant documents to construct accurate, context-grounded answers.Its four-stage pipeline comprises query expansion, diversification, document retrieval, and answer synthesis.Source: §1 · pp. 1–2
  • In certain specialized domains, SearchInstruct improves model performance and instruction-following accuracy, and it enables efficient model updates using newly acquired knowledge.The approach is also reported to increase SFT data quality and diversity.Source: §1 · pp. 1–2

2 Related Work

Prior instruction-tuning and retrieval approaches address scale, diversity, alignment, or factual grounding, but the paper identifies limits for domain-specific adaptation and ongoing updates. SearchInstruct differs by retrieving documents during dataset construction, grounding both instructions and responses in current domain-specific sources.

2 Related Work

  • Wang et al. (Wang et al., 2022b)'s Super-NaturalInstructions and InstructGPT (Ouyang et al., 2022) broaden task coverage or align preferences, yet remain static and costly to construct.The paper also notes they may be insufficient for domain-specific applications or ongoing model updates.Source: §2 · p. 2
  • Self-Instruct (Wang et al., 2022a) bootstraps instructions from a small seed set, but relies on internal knowledge that may be outdated or incomplete for specialized domains.Source: §2 · p. 2
  • Alpaca, Evol-Instruct by Xu et al. (Xu et al., 2023), and InstructZero (Chen et al., 2023) automate instruction generation but lack real-world grounding and domain-specific adaptation.Source: §2 · p. 2
  • RAG (Lewis et al., 2020), REALM (Guu et al., 2020), InstructRetro (Wang et al., 2024), and RA-DIT (Lin et al., 2024) use retrieval, but can require costly retriever training.SearchInstruct instead retrieves during dataset construction, grounding both instructions and responses in current domain-specific documents for fine-tuning.Source: §2 · p. 2

3 Methodology

SearchInstruct constructs SFT instruction–response data by diversifying questions and grounding generated answers in retrieved context. Its four stages are seed generation, query expansion, document retrieval, and response construction.

3.1 SearchInstruct Dataset Formulation

  • The final dataset pairs seed and expanded instructions with answers grounded in retrieved context, while training and inference do not require that context.The retrieved context is used during dataset construction so the model can internalize external knowledge through SFT.Source: §3.1 · p. 3

3.2 Stage 1: Seed Generation

  • Annotators create seeds manually or with LLM assistance, selecting or editing generated queries; this yielded approximately twice as many seeds per annotator as fully manual work.The process starts by generating domain subtopics and then instruction-style queries for each subtopic.Source: §3.2 · p. 4

3.3 Stage 2: Query Expansion

  • At each expansion step, an LLM generates queries from selected seeds as paraphrases or novel combinations, repeating the process over multiple iterations.The expansion adapts Self-Instruct to generate instructions rather than complete instruction–response pairs.Source: §3.3 · p. 4

3.4 Stage 3: Document Retrieval

  • The pipeline retrieves relevant domain evidence using document retrieval, web search, or external APIs, and rewrites instructions as search queries to reduce ambiguity.The rewrite is intended to better match the retrieval source's structure and vocabulary.Source: §3.4 · pp. 4–5

3.5 Stage 4: Response Construction

  • Before answer generation, retrieved content is filtered for relevance or noise, then combined with the instruction to produce a context-grounded response.Filtering can use LLM-based chunk selection or rules that remove noise such as HTML tags, ads, metadata, and comments.Source: §3.5 · p. 5

4 Applications of SearchInstruct

Section 4 applies SearchInstruct to Iranian-culture SFT and updating model knowledge with current information. Across these settings, targeted instruction data addresses domain-specific gaps, while model editing is evaluated for both updated responses and retained general knowledge.

4 Applications of SearchInstruct

  • SearchInstruct is applied to Iranian-culture SFT and to updating model knowledge with up-to-date information for recent or evolving queries.Source: §4 · p. 5

4.1 SFT Dataset Construction for Iranian Culture

  • Iranian-culture SFT addresses underrepresented cultural details and limited access to real-world Persian queries, including colloquial, context-rich requests.Source: §4.1 · pp. 5–6

4.1.1 Improvements in Cuisine and Tourism

  • SearchInstruct-trained models performed significantly better on previously weak areas, and updated responses were correct and current without a significant MMLU accuracy drop.The human evaluation compared baseline and fine-tuned models across two domains; the general evaluation assessed whether model editing degraded broader knowledge.Source: Figure 2 · p. 6, §4.1.1 · p. 6, Table 2 · p. 8
  • After evaluation identifies missing query types, the pipeline generates targeted instruction–response pairs and refines later stages to address prior-round gaps.The reported gaps included requests for multiple examples, scenarios, or recommendations based on personal constraints.Source: §4.1.1 · p. 6

4.2 Model Updating via SearchInstruct

  • For model updating, retrieved current documents guide minimal revisions to selected response spans, leaving unrelated content unchanged.The evaluation focused on a narrow scope of recent political changes and selected current events.Source: §4.2 · p. 7

4.2.1 Evaluation of the Edited Model

  • Model editing can update specific facts without deeply integrating related knowledge: the edited model sometimes lacked background information absent from the update data and its prior knowledge.Source: §4.2.1 · p. 8

5 Conclusion

SearchInstruct combines LLM-driven instruction generation with targeted document retrieval to build SFT datasets. The paper reports gains in cultural-domain model performance and factual updates through minimally invasive editing.

5 Conclusion

  • SearchInstruct combines LLM-driven instruction generation with targeted document retrieval to construct high-quality instruction datasets for SFT.The framework is presented as addressing diversity, realism, and recency challenges, especially in specialized domains.Source: §5 · p. 8
  • The method was evaluated for cultural-domain performance enhancement and factual-knowledge updates through minimally invasive model editing.The domains named include Iranian cuisine and tourism.Source: §5 · p. 8
  • The authors describe SearchInstruct as a flexible, scalable solution for data construction and model editing.The passage presents it as a promising tool for adaptive, domain-aware model development.Source: §5 · p. 8

6 Limitations

The limitations concern retrieved-document reliability, editing depth, resource demands, LLM quality, and bias propagation. The appendices describe dedicated prompts for each pipeline stage and examples of responses improved with supporting documents.

6 Limitations

  • Answer quality depends on reliable retrieved documents; sparse or noisy sources can limit performance, and web-source biases can affect neutrality and fairness.The framework also depends on high-quality LLMs for query expansion and response construction, so performance may degrade in low-resource settings.Source: §6 · p. 9
  • Surface-level answer substitution may not integrate concepts deeply, while large or fast-changing domains require substantial retrieval, filtering, and validation resources.The authors warn that models may memorize updates without understanding broader context.Source: §6 · p. 9

Appendix A System Prompt Designs

  • Each pipeline stage uses a dedicated system prompt tailored to that component’s objective.The appendix presents these designs as structured illustrations.Source: Appendix A · p. 11

A.1 System Prompt for Query Expansion

  • The query-expansion prompt asks the model to generate diverse, semantically rich variations of an initial instruction seed.Its design is illustrated in Figure 5.Source: §A.1 · p. 11

A.2 System Prompt for Search-Oriented Rewriting

  • The search-oriented rewriting prompt transforms general instructions into search-optimized queries for document retrieval.The rewriting aims to match the retrieval source’s structure and vocabulary while reducing ambiguity.Source: §A.2 · p. 11

A.3 System Prompt for Response Construction

  • The response-construction prompt guides the model to synthesize a grounded, context-aware answer from retrieved documents.Source: §A.3 · p. 11

A.4 System Prompt for Answer Updating

  • The answer-updating prompt is designed to make local edits when new or corrected information is introduced.Source: §A.4 · p. 11

Appendix B Failure Cases of Current LLM Models

  • Examples compare original model responses with revised answers produced using the SearchInstruct pipeline; the authors say relevant supporting documents can improve specialized-domain performance.The examples are presented in Table 3 for cases where the base model fails to respond reliably.Source: Appendix B · p. 11

Appendix C Seeds Generation

Seed construction combines expert-written questions with human–LLM collaboration, with human review used to maintain clarity and domain relevance. The culinary seeds target realistic cooking challenges and diverse topics.

Appendix C Seeds Generation

  • The seed-generation process uses both fully manual and human–LLM collaborative strategies, with human selection and editing of outputs against quality criteria.The collaborative setup is reported to accelerate idea generation while supporting controlled diversity and linguistic variation under human oversight.Source: Appendix C · pp. 11–12

C.1 Culinary Domain

  • For culinary seeds, the two strategies aim to capture realistic cooking challenges that require reasoning beyond standard recipes while maintaining topical diversity.Source: §C.1 · p. 12

C.1.1 Human-Crafted Seeds

  • Human-crafted culinary questions focus on mistakes, decisions, ingredient constraints, and reasoning, with more than 30 thematic categories offered as inspiration rather than limits.The categories include food preservation, regional dishes, dietary limitations, cooking science, and ingredient reuse.Source: §C.1.1 · p. 12

C.1.2 Human–LLM Collaborative Seeds

Human–LLM collaboration supports culinary seed ideation and refinement, while tourism seeds target travelers’ complex needs through manual and collaborative construction. Tourism questions cover varied travel constraints and reasoning challenges.

C.1.2 Human–LLM Collaborative Seeds

  • Annotators used LLMs to generate candidate questions, rewrite drafts, and suggest prompts; human reviewers refined outputs for clarity, relevance, and natural phrasing.The authors report improved topical coverage and linguistic diversity alongside faster seed creation.Source: §C.1.2 · pp. 12–13

C.2 Tourism Domain

  • Tourism seeds use both human-authored and human–LLM collaborative construction to represent experiential, cultural, seasonal, and logistical travel needs in Iran.Source: §C.2 · p. 13

C.2.1 Human-Crafted Seeds

  • Manual tourism questions simulate authentic concerns such as destination comparisons, cultural experiences, infrastructure, weather, and cost; the examples include multidimensional reasoning across preferences and logistics.The authors also state that some SearchInstruct responses outperform larger models on selected instructions when given relevant contextual information.Source: §C.2.1 · pp. 13–14, Table 3 · p. 14

C.2.2 Human–LLM Collaborative Seeds

The appendix describes human–LLM collaboration for creating diverse tourism instruction seeds and a separate set of seeds targeting outdated model knowledge. The latter supports both model correction and temporal-generalization evaluation.

C.2.2 Human–LLM Collaborative Seeds

  • Annotators prompted LLMs to generate, refine, or extend questions, then reviewed outputs for realism, clarity, and tone.This supported exploration of lesser-covered subdomains, stylistic variation, and faster seed creation when human inspiration plateaued; together, the methods produced seeds covering typical and edge-case tourism needs.Source: §C.2.2 · p. 14

C.3 Model Updating Domain

  • Model-updating seeds target questions where factual answers may have changed since training, including public figures, international events, institutions, and sports.The seeds are intended for fine-tuning or model editing and as a targeted temporal-generalization evaluation tool.Source: §C.3 · p. 15
  • The model-updating dataset serves as both an editing benchmark and fine-tuning input for time-sensitive factual correction.Its questions aim to surface temporal-awareness blind spots and provide entry points for knowledge injection.Source: §C.3 · p. 15

Appendix D Examples from the Updated Dataset

The appendix illustrates a model-updating dataset built around time-sensitive questions and paired rejected and chosen responses. Chosen answers make targeted factual corrections while preserving the original response as much as possible.

Appendix D Examples from the Updated Dataset

  • Each dataset entry pairs a time-sensitive question with a rejected response and a chosen response.Rejected responses are outdated, incorrect, or uncertain; chosen responses are minimally edited and factually updated.Source: Appendix D · p. 15
  • Chosen responses replace only the incorrect portion, preserving the original answer’s structure, phrasing, and tone where possible.The design targets localized corrections through fine-tuning or model editing rather than general retraining.Source: Appendix D · p. 15
  • Table 4 shows examples spanning elections, leadership updates, international conflicts, and sports.Its rows pair a real-world question with an outdated or incorrect response and a minimally revised accurate version.Source: Appendix D · p. 15

Appendix E – Training Hyperparameters

The appendix summarizes hyperparameters across domain-specific SFT and ORPO training configurations. It identifies the models and datasets involved and the categories of settings covered.

Appendix E – Training Hyperparameters

  • The SFT configurations cover domain-specific culinary and tourism instruction datasets using Matina-8B and Matina-70B.The appendix lists these as separate SFT configurations.Source: Appendix E · p. 15
  • The ORPO configuration trains Gemma-3-27B-IT with LoRA adapters and a custom ORPO objective.Source: Appendix E · p. 15
  • Table 5 provides a unified summary of the hyperparameters, covering LoRA configuration, learning rate, dropout rate, batch size, training duration, and preference-optimization parameters where applicable.Source: Appendix E · p. 15

Appendix F: Future Work

The future-work discussion identifies areas for extending SearchInstruct, including deeper retrieval analysis, automated refinement, structured knowledge integration, and automatic training-data assessment. It also proposes using agent interactions as trajectories for reinforcement-learning training.

Appendix F: Future Work

  • Future work could evaluate how RAG configurations, external API sources, and retrieval strategies affect data diversity and factual accuracy across domains.The current pipeline only briefly references RAG techniques and external APIs, without thoroughly analyzing their practical impact.Source: Appendix F · p. 16
  • Multi-step agent interactions for retrieval, source evaluation, answer synthesis, and refinement could provide trajectories for reinforcement-learning training.The proposed direction targets more capable agents in real-world information-seeking scenarios.Source: Appendix F · p. 16
  • Combining retrieved text with knowledge graphs, ontologies, or semantic lexicons could improve precision and coherence in knowledge-intensive domains.The passage names law, medicine, and education as examples of such domains.Source: Appendix F · pp. 16–17
  • The current iterative feedback loop relies on human feedback and manual adjustments; future work could optimize it using reinforcement learning or other methods.The proposed optimization would refine query generation, retrieval, and synthesis based on performance signals.Source: Appendix F · p. 17
  • Automatic metrics or learned models could assess evidential coverage, factual accuracy, and stylistic consistency to reduce reliance on manual review.The paper identifies generated-data quality evaluation as a bottleneck.Source: Appendix F · p. 17
Loading 2509.10708v1…