SAIL: Search-Augmented Instruction Learning

• LLMs may lack up-to-date knowledge because their fixed training data cannot incorporate later information without costly retraining.Source: §1 · p. 1
• LLM generations are opaque and can produce misinformation, stereotypes, or toxic language without trustworthy sources.Source: §1 · p. 1
• Commercial search engines provide current information but can return noisy or distracting passages that mislead language models.Source: §1 · pp. 1–2
• SAIL trains a model to generate responses from noisy search results while learning to denoise retrieval and ground on useful information.Source: §1 · p. 2
• The work constructs a search-augmented instruction corpus and fine-tunes SAIL-7B, which outperforms GPT-3.5-Turbo and Vicuna-13B on several NLP tasks.Source: §1 · p. 2
• Across instruction following, question answering, and language checking, SAIL-7B is robust to distracting retrieval and reaches performance comparable to state-of-the-art instruction-following models.Source: §1 · pp. 2–3

2.1 Search Result Collection

• Queries combine each instruction with its input, and DuckDuckGo plus BM25 Wikipedia retrieval supply retained search-result titles and previews.Source: §2.1 · p. 3
• Training examples pool five retrieved passages and randomly sample zero to three results, exposing the model to varied retrieval sources and quantities.Source: §2.1 · p. 3

2.2 In-context Retrieval Selection

• An entailment model labels a passage informative when its entailment score exceeds its contradiction score; otherwise it is labeled distracting.Source: §2.2 · p. 3

2.3 Fine-tuning

• Prompts place the most relevant retrieval nearest the instruction, and both retrieval-selection sequences and annotated responses are used for fine-tuning.Source: §2.3 · p. 3

2.4 Evaluation

• Instruction-following evaluation compares model responses with GPT-4 using scores on 80 Vicuna-Instructions questions, normalized by GPT-4 scores.Source: §2.4 · pp. 3–4
• Question answering uses CSQA, OBQA, and ARC-Challenge in instructed zero-shot and search-augmentation settings to test sensitivity to noisy retrieval.Source: §2.4 · p. 4
• Fact and fairness checking uses UniLC tasks in zero-shot and search-augmented settings, comparing Vicuna-7B, Vicuna-13B, and SAIL-7B with and without search.Source: §2.4 · p. 4

3.1 Instruction Following

• Search-augmented SAIL-7B scores 90% versus below 85% for other models on instruction following, while using fewer training instructions and parameters.It achieves 103% of ChatGPT performance with search and 98% without search, although GPT-4 remains better.Source: §3.1 · p. 4
• Search augmentation improves SAIL-7B from 84% to 90% and from 98% to 103% across the two instruction-following evaluations, whereas it can hurt Vicuna and LLaMA-GPT4 models.The search results contain useful information, but models without search-augmented fine-tuning can be misled by distracting retrieval results.Source: §3.1 · p. 5

3.2 Question Answering

• DuckDuckGo produces better question-answering performance than Wikipedia retrieval across models and tasks, but searched-augmented LLaMA-7B and Vicuna-7B average below 50% accuracy.DuckDuckGo covers a wider range of information, while its results can still include unrelated or distracting passages.Source: §3.2 · p. 7
• Vicuna-13B achieves 51.0% average accuracy without grounding, but search results hurt its accuracy in most experiments, including a 1.4% decrease on ARC-Challenge.Wikipedia BM25 results decrease its accuracy by as much as 1.8%, showing that a larger baseline does not reliably exploit retrieved evidence.Source: §3.2 · p. 7
• SAIL-7B improves on all three question-answering tasks with search results and reaches 1.2% higher average accuracy, including gains of 2.8% on OBQA.Without retrieval, SAIL-7B averages 49.5%, versus 44.5% for LLaMA-7B and 40.9% for Vicuna-7B; search-augmented SAIL-7B performs best on CSQA and OBQA.Source: §3.2 · pp. 7–8

3.3 Fact and Fairness Checking

• SAIL-7B achieves the highest accuracy and F1 scores on all four UniLC fact and fairness tasks without grounding, while search augmentation improves its fact-checking results on both evaluated tasks.On Climate-Fever, the improvement reaches 4.2%, and augmented SAIL outperforms baselines by 9% accuracy and 5% F1.Source: §3.3 · p. 8

4.1 Capabilities

• Large language models exhibit zero-shot multitask solving and few-shot in-context learning with chain-of-thought reasoning.These capabilities provide the broader language-model context for instruction following and retrieval augmentation.Source: §4.1 · p. 8
• Instruction tuning improves zero-shot task performance and enables models such as InstructGPT, ChatGPT, and GPT-4 to handle diverse tasks without demonstrations.Research also explores generated instructions and instruction-following outputs to improve this capability.Source: §4.1 · pp. 8–9
• Retrieval-augmented language models have used end-to-end retriever training, frozen retrievers, dedicated losses, or frozen language models with tuned retrieval modules.Examples include REALM, RAG, RETRO, Atlas, RePlug, and In-context RALM.Source: §4.1 · p. 9
• SAIL addresses prior retrieval-augmented models’ closed-corpus, distraction, and limited instruction-tuning constraints by using real-world search engines, search-result denoising, and instruction fine-tuning.These changes support broader applications including chatbot instruction following, fact checking, and fairness checking.Source: §4.1 · p. 9

4.2 Trustworthiness

• Self-improvement methods use chain-of-thought, self-consistency, in-context examples, self-reflection, and self-refinement to improve reasoning or adaptation.These approaches concern model-generated reasoning and refinement rather than SAIL’s search-result denoising process.Source: §4.2 · p. 9
• Fact and fairness checking research examines whether language models can identify hallucinations and assess harmful language produced by humans or machines.This motivates trustworthiness-oriented evaluation alongside response generation.Source: §4.2 · p. 9

5 Conclusion

• The paper constructs a search-augmented instruction-tuning corpus combining human instructions, GPT-4 responses, and BM25 and commercial-search results, then fine-tunes LLaMA-7B.The corpus is designed to address disputed and distracting retrieval results during generation.Source: §5 · p. 9
• SAIL-7B distills trustworthy and helpful information from search results and generates high-quality responses across instruction following, question answering, and fact/fairness checking.The reported outcome is improvement in both performance and transparency for instruction-following language models.Source: §5 · pp. 9–10

Limitations

• SAIL’s major limitation is that it does not explain why a search result is trustworthy or informative.Future work proposes larger models that recognize trustworthy search results with explanations.Source: Limitations · p. 10