PaperlayerView exact revision on arXiv

Source-linked AI summary

TriageRAG: Failure-Aware Self-Training for Adapting Retrieval-Augmented Language Models to Specialized Domains

Samarth Varshney, Ritu Yadav

arXiv:2610.33776v1cs.CLcs.AIcs.IRcs.LG

TL;DR

Specialized-domain RAG adaptation is difficult when labeled QA data is scarce, and existing self-training does not distinguish what the student knows from reader or retrieval failures. TriageRAG probes each self-generated example and routes it by failure type; it outperforms budget-matched SimRAG across the reported domains and backbone settings.

  • Problem

    Adapting RAG to specialized domains is difficult when reader and retriever face distribution shift and labeled domain QA data is scarce or private.

    Source: §1 · p. 1

  • Method

    TriageRAG probes each self-generated question with and without retrieved context, classifies the student’s behavior into four categories, and routes each category to a training signal.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    TriageRAG outperforms budget-matched SimRAG by 1.2–6.4 points across three domains and two backbone sizes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The ablation identifies the parametric route as the ingredient behind the gap: round-trip filtering removes contexts without evidence, and readers that never see them over-trust noisy retrieval.

    Source: §5.2 · p. 7, §6 · p. 9

    The pool is synthesized only once, so informative examples are consumed early and accuracy plateaus; regenerating the pool between rounds is future work.

    Source: §7 · p. 9

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) lets large language models (LLMs) answer knowledge-intensive questions from retrieved evidence, but adapting a general-purpose RAG system to a specialized domain such as medicine is difficult when only an unlabeled corpus is available. Self-training approaches such as SimRAG address this by letting the LLM write its own question-answer pairs from domain passages and fine-tuning on the pairs that survive a round-trip consistency filter. We observe that these approaches treat every retained example identically: they never ask whether the student already knows the answer, already extracts it correctly from the retrieved context, or fails - and if so, whether the failure lies with the reader or with the retriever. To tackle this, we propose TriageRAG, a failure-aware self-training approach that probes the student on each self-generated question with and without retrieved context, triages it into one of four categories (known, solved, context-failure, retrieval-failure), and routes each category to the training signal that addresses it, including a parametric route that teaches the reader to answer when retrieval misses. Probing is cheap and repeated every round, so the training mixture follows the student, and the triage histogram doubles as a label-free diagnostic of where the domain gap lies. Experiments on 11 datasets spanning three domains with a Qwen3-4B backbone, and on CS-Bench with Qwen3-8B, demonstrate that TriageRAG outperforms a budget-matched SimRAG by 1.2-6.4 points and is the only method that recovers the closed-book level in every case where retrieval hurts the base model. A single ablation identifies the mechanism: round-trip filtering removes every training context that lacks the evidence, and a reader that never sees such contexts learns to over-trust noisy retrieval.

1 INTRODUCTION

Adapting RAG to specialized domains is difficult with scarce labeled data, and existing self-training treats retained examples alike despite differences in what the student knows and what retrieval supplies. TriageRAG probes each example, routes it by failure type, and improves on budget-matched SimRAG across the reported domains and backbones.

1 INTRODUCTION

  • RAG adaptation is difficult when reader and retriever face domain shift and labeled domain QA data is scarce or private.SimRAG instead synthesizes questions from unlabeled passages and retains pairs whose source passage is retrieved in the top-k.Source: §1 · p. 1
  • Existing self-training treats retained examples alike, although the student may already know the answer, fail to use present evidence, or receive no evidence.Round-trip filtering removes examples whose evidence is absent, so the reader does not train on contexts lacking the answer and may learn to trust noisy retrieval.Source: §1 · pp. 1–2
  • TriageRAG probes each self-generated pair with and without retrieved context, assigns one of four failure categories, and routes it to a matching training signal.It repeats the inexpensive diagnosis each round; the category histogram also provides a label-free diagnostic of the domain gap.Source: §1 · p. 2
  • Across three domains and two backbone sizes, TriageRAG, trained with SFT only and no reward model, judge, or labeled data, outperforms budget-matched SimRAG by 1.2–6.4 points; headline comparisons use three training seeds and paired bootstrap intervals.Source: §1 · p. 2

2 RELATED WORK

Related work spans RAG readers and domain adaptation, self-training, and knowledge-aware fine-tuning. TriageRAG combines self-generated data with failure-type diagnosis to select training signals for a corpus-grounded RAG system.

Retrieval-augmented generation

  • RAFT trains readers using GPT-4-written answers with an oracle document and distractors, while Schimanski et al. (2024) distill context-aware QA data from GPT-4.SimRAG removes the proprietary teacher by developing question-answering and question-generation skills in the student and retaining round-trip-consistent pairs Bartolo et al. (2021).Source: §2 · pp. 2–3

Self-training and self-improvement

Knowledge-aware fine-tuning

  • Knowledge-aware fine-tuning work uses labeled data; TriageRAG applies knowledge probing to self-generated data and distinguishes whether evidence was retrieved.Prior work includes weighting examples by conflict between parametric and contextual knowledge and studying when to trust memory over context.Source: §2 · p. 3

3 METHODOLOGY

TriageRAG adapts a Stage-I reader to an unlabeled domain corpus by diagnosing each self-generated tuple and routing it to a corresponding training signal. The main configuration freezes the retriever.

3.1 PROBLEM SETUP

  • The main setup adapts the LLM’s QA capability using an unlabeled corpus, with Stage-I retrieval-oriented fine-tuning followed by Stage-II self-generated pseudo-labeled tuples.The retriever is frozen in the main configuration.Source: §3.1 · p. 3

3.2 STAGE-I: RETRIEVAL-ORIENTED FINE-TUNING

  • Stage-I fine-tunes the instruction-tuned backbone on 16,000 general retrieval-oriented examples, including context-aware QA and answer- and question-generation skills.The latter two skills are used during synthesis.Source: §3.2 · p. 3

3.3 STAGE-II: FAILURE-AWARE SELF-TRAINING

  • Stage-II synthesizes self-answerable question-answer-passage tuples once, then probes the student with and without top-k retrieved passages and records answer pass rates and evidence hits.Triage assigns KNOWN, SOLVED, CONTEXT-FAILURE, or RETRIEVAL-FAILURE using threshold τ=0.75; the category histogram diagnoses the domain gap.Source: §3.3 · pp. 3–4
  • Per-round quotas prioritize context failures, retrieval failures, solved items, and known items; each category receives a distinct training treatment.Context failures receive contextual SFT, retrieval failures receive parametric SFT on contexts without evidence, and solved and known items receive small consolidation shares plus general replay.Source: §3.3 · p. 4

3.4 A RETRIEVER ROUTE

  • The retriever route contrastively fine-tunes the dense retriever on retrieval failures, using the source passage as a positive and a top-ranked passage as a hard negative.This variant degrades retrieval and accuracy, so the main configuration keeps the retriever frozen.Source: §3.4 · p. 4

4 EXPERIMENTAL SETUP

The evaluation covers 11 datasets across medicine, science, and computer science, with shared-budget baselines and a common training and retrieval setup. The reported implementation uses Qwen3 backbones and three TriageRAG rounds.

4.1 TASKS AND DATASETS

  • Evaluation spans medical, scientific, and computer-science datasets, with accuracy as the metric across multiple-choice and yes/no question formats.The medical set includes MIRAGE datasets and MedXpertQA; science includes ARC-Challenge, SciQ, and scientific MMLU subsets; CS uses CS-Bench questions.Source: §4.1 · pp. 4–5

4.2 BASELINES

  • Stage-II comparisons share the Stage-I model, synthetic pool, self-answerability filter, budget, replay, and training recipe, varying example selection and context construction.Equal-budget baselines include SimRAG, random selection, and a RAFT-style mixture; the study excludes GPT-4-distilled readers and domain LLMs trained on overlapping task data.Source: §4.2 · p. 5

4.3 IMPLEMENTATION DETAILS

  • The experiments use Qwen3-4B and Qwen3-8B, a bge-base-en-v1.5 retriever with k=5, and three TriageRAG rounds.Each domain has a specified per-round budget; evaluation is zero-shot with greedy decoding, and one medical round on 4B takes about 15 minutes of probing and 80 minutes of LoRA training.Source: §4.3 · p. 5

5 EXPERIMENTAL RESULTS

TriageRAG outperforms budget-matched SimRAG across the evaluated domains and backbones. The experiments link this advantage to training on evidence-absent contexts, examine what the triage reveals, and test robustness across seeds.

5.1 MAIN RESULTS

  • Against budget-matched SimRAG, TriageRAG gains 1.2 points in medicine, 1.9 in science, 6.4 on CS-Bench (4B), and 4.4 (8B).It is the best fine-tuned method on average in every domain and best or second best on 8 of 11 datasets.Source: §5.1 · p. 5, Table 1 · p. 6, Table 2 · p. 6
  • Where five retrieved passages hurt the backbone, TriageRAG alone returns to or exceeds its closed-book level in all three reported cases: 87.6 vs. 87.4, 68.2 vs. 68.0, and 70.1 vs. 69.8.The last comparison is Qwen3-8B; the first two are the science and CS cases reported for Qwen3-4B.Source: §5.1 · p. 5

5.2 ABLATION STUDIES

  • Replacing the parametric route with gold-passage contexts drops TriageRAG from 68.2 to 61.9 on CS-Bench, 87.6 to 84.3 in science, and 64.8 to 63.7 in medicine.Each ablated result is at or near the corresponding SimRAG level, supporting the importance of training on contexts without the evidence.Source: §5.2 · p. 7, Table 4 · p. 7

5.3 STUDY ON THE TRIAGE

  • About 41% of generated questions in medicine and CS and 57% in science are KNOWN; RETRIEVAL-FAILUREs make up 29% of the medical pool versus 7% in science and CS.In CS and science, retrieval finds evidence for 84–89% of questions; CONTEXT-FAILUREs comprise 4–6% of the pool.Source: §5.3 · p. 8
  • Mixtures with at most 2% evidence-absent contexts stay at or below Stage-I, while those with 10% or more reach within a point of closed-book accuracy or above.The comparison spans round-1 Stage-II mixtures; the ablation without the parametric route, like SimRAG, has at most 2% such examples.Source: §5.3 · p. 8, Figure 2 · p. 8

5.4 ROBUSTNESS: TRAINING SEEDS AND CONFIDENCE INTERVALS

  • Across three seeds, TriageRAG beats SimRAG in every domain, by +5.1 points on CS, +1.8 in medicine, and +2.8 in science.TriageRAG also beats the ablation without the parametric route on every seed in every domain; seed standard deviations range from 0.1 to 1.2.Source: §5.4 · pp. 8–9

6 CONCLUSION

TriageRAG adapts retrieval-augmented LLMs to specialized domains from an unlabeled corpus by probing students on self-generated questions and routing examples by failure type. Across 11 datasets and two backbone sizes on CS-Bench, it outperforms budget-matched SimRAG; the paper attributes the mechanism to training on contexts without evidence.

6 CONCLUSION

  • TriageRAG probes each self-generated question with and without context, then routes examples by failure type to the matching training signal.The framework adapts retrieval-augmented LLMs to specialized domains using an unlabeled corpus.Source: §6 · p. 9
  • Across 11 datasets in three domains and two backbone sizes on CS-Bench, TriageRAG consistently outperforms budget-matched SimRAG.The paper describes a single ablation as explaining why: round-trip filtering removes contexts lacking evidence, and a reader that never sees such contexts over-trusts noisy retrieval.Source: §6 · p. 9
  • The triage also serves as a label-free diagnostic of where the domain gap lies.Making retrieval failures useful to the retriever itself remains open.Source: §6 · p. 9

7 LIMITATIONS

The authors identify limits in the fixed synthesis pool, model and data choices, and evaluation reporting. The paper also describes its AI-assistance and planned reproducibility materials.

7 LIMITATIONS

  • Re-probing a pool synthesized once causes informative examples to be consumed early and accuracy to plateau; regenerating the pool between rounds is future work.The reported backbones, 16k-example Stage-I blend, and PubMed slice make absolute numbers incomparable with SimRAG’s larger models and fuller data; MedXpertQA is at chance across configurations.Source: §7 · p. 9

AI USE STATEMENT

  • The authors used LLMs for prose and coding assistance, reviewed the resulting text and code, and take responsibility for the final content.They state that experimental designs, numbers, and claims came from their code and were checked against result artifacts.Source: AI USE STATEMENT · p. 9

REPRODUCIBILITY STATEMENT

  • The authors state that they will release code, exact job lists, and per-example predictions for every reported run.They say every number in the paper is generated from summary files by one script.Source: REPRODUCIBILITY STATEMENT · p. 9

A STAGE-I DATA

Stage-I uses a uniform 16,000-example sample from a 20,930-example general blend, covering context-aware QA, answer-span generation, and question generation.

A STAGE-I DATA

  • The 20,930-example blend contains context-aware QA, answer-span generation, and question generation data.Its context-aware QA component draws on SQuAD, BoolQ, and OpenBookQA; the other components draw on SQuAD and Natural Questions.Source: A STAGE-I DATA · p. 13
  • The Stage-I training set is a uniform sample of 16,000 examples from that blend, and the authors omit general instruction-following data because the backbones are already instruction-tuned.The blend mirrors three components of SimRAG’s Stage-I data.Source: A STAGE-I DATA · p. 13
  • One LoRA epoch took 21 minutes on four A100s for the 4B backbone and 67 minutes on two A100s for the 8B backbone.Each Stage-II round also uses 2,000 general replay examples drawn from the same blend.Source: A STAGE-I DATA · p. 13

B TEST DATA DETAILS

The appendix specifies the evaluation datasets and corpora, then details synthesis, baseline construction, TriageRAG’s realized mixture, and retriever tuning.

B TEST DATA DETAILS

  • Evaluation covers medical, scientific, and computer-science datasets, including medical exam and biomedical QA benchmarks, science multiple-choice tests, and CS-Bench subsets.The authors omit CS-Bench fill-in-the-blank and open-ended subsets because they require an LLM judge.Source: B TEST DATA DETAILS · pp. 13–14
  • The medical corpus combines textbook snippets and PubMed abstracts, science uses SciQ and OpenBookQA support passages, and CS uses chunked Wikipedia passages.The resulting corpora contain 950,746 medical passages, 10,863 science passages, and 6,237 computer-science passages.Source: B TEST DATA DETAILS · p. 14

C BASELINE AND SYNTHESIS DETAILS

  • Synthesis filters passages and answer spans, generates questions and distractors, and produces self-answerable pools of 40,540 medical, 32,686 science, and 20,669 CS tuples.The pools are drawn from 12,000 medical, 8,000 science, and 5,089 CS passages, respectively.Source: C BASELINE AND SYNTHESIS DETAILS · p. 14
  • SimRAG keeps tuples whose source passage is among the top five retrieved, while random selection samples uniformly and RAFT-style training uses gold passages for 80% of examples.The RAFT-style remaining 20% uses retrieved passages other than the gold passage; its targets are short answers.Source: C BASELINE AND SYNTHESIS DETAILS · p. 14
  • TriageRAG’s round-one mixture uses 60/20/15/5% quotas for context-failure, retrieval-failure, solved, and known examples, plus 2,000 general replay examples.Training contexts lacking evidence comprise 22.0% in medicine, 20.7% in science, and 20.6% in CS for TriageRAG.Source: C BASELINE AND SYNTHESIS DETAILS · p. 14
  • Retriever tuning fine-tunes bge-base-en-v1.5 on retrieval-failure pairs with a multiple-negatives ranking loss, then re-embeds and reindexes the corpus.The training uses in-batch negatives and one mined hard negative per pair.Source: C BASELINE AND SYNTHESIS DETAILS · p. 14

D ADDITIONAL ANALYSES

The additional analyses examine seed stability, category-level examples, retriever tuning, and differences among medical datasets.

D ADDITIONAL ANALYSES

  • Across every seed, TriageRAG beats SimRAG and the ablation without the parametric route, while both method pairs remain within two points of each other.The three-seed comparison shares the Stage-I model, pool, and round-1 probe.Source: Table 5 · p. 15
  • The case studies illustrate all four triage categories, including retrieval failures that SimRAG’s filter discards.They contrast a known textbook fact, a context-solved statistic, a context failure with a competing passage, and an unretrieved rare complication.Source: D ADDITIONAL ANALYSES · pp. 14–15
  • Fine-tuning the retriever on retrieval-failure pairs degrades both source-passage recall and downstream accuracy in the controlled medical comparison.Source: Table 7 · p. 16
  • In medicine, TriageRAG’s gains over Stage-I appear on exam-style datasets, while literature yes/no datasets move by at most 1.3 points under the listed variants.Random selection and RAFT-style training lower PubMedQA by 4.4 and 3.0 points, respectively; retriever tuning damages both literature datasets.Source: Figure 3 · p. 16

E ADDITIONAL RESULTS

Across rounds, re-probing changes TriageRAG’s training mixture, but accuracy generally moves little after the first round on a fixed pool.

E ADDITIONAL RESULTS

  • Medicine plateaus at 64.8, 64.7, and 64.5 accuracy; science peaks at 88.0 in round two, and CS-Bench peaks at 68.8 in round two.The reported per-round sequences are medicine 64.8, 64.7, 64.5; science 87.6, 88.0, 87.5; and CS-Bench 68.2, 68.8, 68.3.Source: E ADDITIONAL RESULTS · p. 16
  • Re-probing shifts TriageRAG’s category mixture as the student changes: medicine’s KNOWN share rises from 40.9% to 45.2%, while CS-Bench CONTEXT-FAILURE falls from 4.4% to 2.7%.Medicine’s RETRIEVAL-FAILURE share falls from 29.2% to 26.8% across three rounds.Source: E ADDITIONAL RESULTS · p. 16, Figure 4 · p. 17
  • With a fixed pool, informative examples are largely consumed in round one; refreshing the pool between rounds is identified as future work.Source: E ADDITIONAL RESULTS · p. 16
  • Unparsable outputs are counted as wrong; fine-tuned models had at most three per dataset, while the untuned retrieval model had 86 on MedMCQA and 32 on CS-Bench MC.The authors note that the untuned model sometimes answers with a sentence rather than a letter, slightly disadvantaging its retrieval rows.Source: E ADDITIONAL RESULTS · p. 16
  • Table 8 presents one medical question per triage category alongside probe outcomes and the route assigned to each example.Source: Table 8 · p. 17

F EXTENDED LIMITATIONS

The extended limitations bound the results by pool, model and corpus scale, evaluation coverage, and experimental replication; prompt details describe the evaluation and synthesis inputs.

F EXTENDED LIMITATIONS

  • Because the pool is synthesized once, informative examples are largely consumed in round one and accuracy plateaus; regenerating it between rounds remains future work.Source: F EXTENDED LIMITATIONS · p. 17
  • With 4B/8B backbones, a 16k-example Stage-I blend, and only a PubMed slice, absolute numbers are not comparable to SimRAG’s 8B/27B models using full PubMed, textbooks, and Wikipedia.The paper says all its comparisons are internal.Source: F EXTENDED LIMITATIONS · p. 17
  • MedXpertQA remains at chance for every trained configuration and cannot distinguish methods at this scale.Source: F EXTENDED LIMITATIONS · p. 18
  • Headline round-one cells use three seeds and paired bootstrap intervals, but later-round, RAFT-style, and retriever-tuned results are single runs; CS-Bench differences of one point or less are ties.Source: F EXTENDED LIMITATIONS · p. 18
  • The parametric route trains the reader to answer with unhelpful context, not to abstain; calibration and unanswerable-question behavior were not measured.Source: F EXTENDED LIMITATIONS · p. 18
  • The negative retriever result covers one contrastive fine-tuning recipe; curated negatives, smaller learning rates, or mixing original training data may reverse it.Source: F EXTENDED LIMITATIONS · p. 18

G PROMPT DETAILS

  • The evaluation prompt instructs the model to use relevant context and supplies retrieved passages, a question, and answer choices when applicable; closed-book probing omits the context block.Source: G PROMPT DETAILS · p. 18
  • Synthesis prompts first elicit candidate answer spans, then generate answer-conditioned questions or supported/refuted factual claims from a passage.Source: G PROMPT DETAILS · pp. 18–19
Loading 2610.33776v1…