PaperlayerView exact revision on arXiv

Source-linked AI summary

When Synthetic Data Hurts: On Catastrophic Forgetting in Skill Retrieval for LLM Agents

Syed Shariyar Murtaza, Yifan Nie, Utkarsh Soni, Eugene Wen, Arvid Frydenlund

arXiv:2609.10750v1cs.IRcs.AIcs.LG

TL;DR

The paper asks whether synthetic-data fine-tuning harms skill retrieval for real and OOD tasks in large LLM-agent skill repositories. It studies regularized fine-tuning on a 34,396-skill benchmark and finds that forgetting mitigation preserves OOD behavior while improving synthetic in-distribution retrieval.

  • Problem

    The paper investigates whether synthetic-data fine-tuning degrades skill retrieval for real-world and OOD tasks, a question with limited evidence at realistic deployment scale.

  • Method

    The authors evaluate real and synthetic supervision over 34,396 skills and compare embedding-anchor, LwF, EWC, and L2-initialization regularization during fine-tuning.

  • Results

    Synthetic supervision improves in-distribution retrieval but can reduce OOD recall from 0.850 to 0.650, while regularization preserves OOD behavior and improves synthetic retrieval.

  • Takeaways & Limitations

    Conservative regularized fine-tuning provides a practical recipe for retaining real and OOD retrieval behavior while improving performance on a new synthetic training distribution.

  • Takeaways & Limitations

    The positively supervised skills cover only a small fraction of the catalogue, and the synthetic and real reward distributions differ substantially with modest inter-judge agreement.

Abstract

from arXiv · show

LLM agents increasingly rely on external skills retrieved at runtime, making skill selection from large repositories a critical challenge. We present a production skill router over 34,396 skills and a large-scale study of skill retrieval using limited real supervision and synthetic data. We found that the synthetic-data fine-tuning improves in-distribution retrieval but it causes catastrophic forgetting on real and out-of-distribution (OOD) data. We evaluate several forgetting mitigation fine-tuning approaches inspired by continual learning, including embedding-anchor regularization, Learning without Forgetting (LwF), Elastic Weight Consolidation (EWC), and L2-initialization. The results show that these approaches not only retain the performance on OOD skills retrieval but also improve the retrieval on synthetic in-distribution skills by 13.98\% for 0.6B Qwen retriever and reranker. Our results provide a practical benchmark and a robust fine-tuning recipe for scarce, multi-positive supervision.

1 Introduction

The paper studies skill retrieval for LLM agents at realistic repository scale, focusing on whether synthetic supervision causes forgetting on real and OOD tasks. It evaluates regularization strategies that aim to preserve existing retrieval behavior while retaining synthetic-data gains.

  • Synthetic supervision improves in-distribution retrieval but can cause catastrophic forgetting, with OOD recall dropping from 0.850 to 0.650 under aggressive fine-tuning.The result shows that gains on synthetic training data do not necessarily transfer to robust deployment behavior.
  • The central question is whether synthetic-data fine-tuning degrades retrieval for real-world or out-of-distribution tasks, especially at realistic deployment scale.
  • The authors evaluate embedding anchors, LwF, EWC, and L2-initialization as forgetting-mitigation approaches under a common fine-tuning recipe.
  • The study covers 34,396 skills and 109 verifier-scored tasks, identifying synthetic-data-induced forgetting as a deployment failure mode.It also introduces a unified benchmark combining real and synthetic supervision.
  • Regularized fine-tuning preserves real-world and OOD behavior while improving in-distribution gains, providing a practical recipe for deployed skill-retrieval systems.

2 Methodology

The methodology builds synthetic and real supervision for skill retrieval, trains compact retrievers and rerankers with LoRA, and applies four regularizers to mitigate forgetting. The regularizers constrain embeddings, parameters, or output similarity behavior relative to a frozen model.

  • The benchmark combines a 34,396-skill catalogue with real task executions and synthetic supervision generated through anchor-skill, paraphrase, and multi-skill task construction.Real executions are harvested in Harbor, while synthetic data are organized into complementary single-positive and multi-positive tracks.
  • Track B creates multi-positive rows from real positives, paraphrased tasks, and skill-first tasks, then pairs each positive skill with seven negatives from retrieval, similarity, BM25, and random sources.
  • The Qwen3-Embedding-0.6B retriever is fine-tuned with LoRA and InfoNCE, with Track B positive examples additionally weighted in the loss.
  • Forgetting mitigation: Embedding-anchor regularization penalizes deviations between fine-tuned and frozen positive-skill embeddings, while L2-init constrains LoRA parameters near initialization.
  • Forgetting mitigation: EWC weights parameter drift by Fisher importance, whereas LwF preserves the frozen model’s similarity distribution through temperature-scaled KL divergence.

3 Experiments

The experiments evaluate retrieval under real, synthetic, and mixed supervision using task-disjoint splits, held-out synthetic skills, and external real tasks. They compare frozen baselines and fine-tuned variants with task-level uncertainty estimates.

  • Track A evaluates Synth-only, Real-only, and Real+synth settings on a 75-task real pool, using the difference between Real-only and Real+synth to isolate synthetic-data benefit.
  • Track B evaluates three disjoint rings: real SKILLSBENCH tasks, held-out synthetic skills, and real TERMINALBENCH 2 tasks.
  • Track B varies LoRA recipes and synthetic-data mixtures, evaluating each model on real and synthetic test sets with different task and skill compositions.
  • The study compares four frozen systems: hybrid retrieval, dense 0.6B retrieval, a 0.6B reranker stack, and BM25.
  • The evaluation reports HIT@k, RECALL@k, MRR, effect sizes, and bootstrap confidence intervals while accounting for task-level variance.

4 Results

Across two synthetic-training tracks, aggressive fine-tuning improves or overfits synthetic retrieval while degrading real and OOD behavior; regularized forgetting mitigation preserves OOD quality and improves synthetic retrieval. The final retriever–reranker stack further improves real-task retrieval through blended scoring.

  • Synthetic-data regression: Track-A synthetic supervision lowers RECALL@10 by 0.026 for synth-only training and by 0.021 for Real+synth versus Real-only.Real-only training matches or slightly improves the baseline, whereas adding synthetic data does not help and reduces recall.
  • Synthetic-data regression: Aggressive Track-B LoRA overfits the synthetic ring and collapses on real OOD rings, motivating regularized fine-tuning.The same catastrophic-forgetting pattern appears in Track A, where aggressive adaptation produces larger negative shifts and task-level regressions.
  • Forgetting mitigation: Conservative forgetting mitigation preserves OOD retrieval while improving in-distribution synthetic retrieval across Track B configurations.All four approaches maintain recall comparable to baseline embedding models on OOD Rings 1 and 3, while improving Ring 2.
  • Forgetting mitigation: Low-rank conservative updates with r = 8 preserve real/OOD performance with minimal seed sensitivity, while anchor weight has only a minor effect.The sweep identifies LoRA rank as the primary factor controlling the performance-retention trade-off.
  • Cross-encoder reranker: On 75 real deployment tasks, reranker blending raises retrieval to h@5 = 0.893 and MRR = 0.781, while LwF reaches h@5 = 0.861 and MRR = 0.744 on synthetic data.The final stack uses anchor-regularized embedding and reranker LoRA models with blended scoring.

5 Related Work

Prior work studies skill retrieval with synthetic supervision and catastrophic forgetting, while this paper examines how preservation strategies behave in synthetic-data-driven skill retrieval for LLM agents.

  • Skills, retrieval, and synthetic supervision for agents: Synthetic supervision is a standard low-supervision recipe for retrieval, but its effect depends on whether generated tasks capture downstream multi-positive structure.The paper reports benefits for lightweight components but possible harm when fine-tuning encoders or rerankers.
  • Continual learning and catastrophic forgetting: Catastrophic forgetting is established in continual learning, where methods such as EWC and LwF preserve prior behavior through parameter or output regularization.The related methods also include L2-init, which keeps parameters close to initialization.
  • Continual learning and catastrophic forgetting: This study applies embedding anchoring, LwF, EWC, and L2-init under a shared conservative LoRA recipe to synthetic-data-induced forgetting in large-scale skill retrieval.The approaches are evaluated in a common empirical setting for LLM agents.

6 Conclusion

The paper studies skill retrieval at 34,396-skill scale and finds that naive synthetic-data fine-tuning can degrade real and OOD retrieval. Forgetting-mitigation regularization preserves OOD behavior while improving in-distribution performance.

  • 6 Conclusion: The unified benchmark combines real tasks and synthetic supervision for skill retrieval across a 34,396-skill repository.The benchmark supports the paper’s study of real-task retrieval and synthetic-data-induced forgetting.
  • 6 Conclusion: Forgetting-mitigation regularizers with LoRA improve in-distribution retrieval while preserving out-of-distribution performance.The conclusion characterizes aggressive-update regression as a stability-plasticity failure consistent with representational or ranking-function drift.
  • 6 Conclusion: Compact 0.6B retrievers can match much larger hybrid systems, highlighting supervision quality as important relative to model scale.This conclusion is reported alongside the forgetting-mitigation findings.

Limitations

The evaluation is limited by small validation sets, imperfect synthetic-data quality and coverage, restricted task scope, and dependence on closed-weight LLMs. These constraints limit how broadly the findings and synthetic supervision can be generalized.

  • Statistical power: Val-set 1 and Val-set 3 are small enough that single-task variation can shift RECALL@10 by approximately 5%.The paper addresses this with paired bootstrap sampling, effect-size analysis, and a large OOD benchmark.
  • Known quality-gate failures in Track B: Track B’s synthetic and real reward distributions differ, positive-skill coverage is sparse, and inter-judge agreement is modest.These issues constrain treating the synthetic mixture as a faithful proxy for real supervision.
  • Scale and coverage of the training signal: Positive supervision covers only a small fraction of the approximately 34K-skill catalogue, so some negative results may change with broader coverage or more synthetic data.The paper specifically notes this uncertainty for encoder LoRA fine-tuning.
  • Scope of domains, language, and modality: All experiments use English, text-based tasks in a dockerized harness and do not evaluate multilingual, multimodal, or non-code agent settings.Transfer of the observed forgetting patterns and mitigations to other domains remains open.
  • Dependence on closed-weight LLMs: Both synthetic pipelines depend on commercial closed-weight LLMs, making exact data and judge-label reproduction sensitive to model or API drift.The paper documents the pipeline, but external endpoints constrain full reproducibility.

Ethics Statement

The study documents licensing, privacy, decontamination, and content checks for its benchmark and synthetic tasks. It also uses leakage probes to separate synthetic and real evaluation data.

  • Ethics Statement: The catalogue and benchmark tasks are used under license, and no personally identifying information is added.This states the paper’s licensing and privacy safeguards.
  • Ethics Statement: Synthetic tasks are decontaminated with a Jaccard reject filter and spot-checked for offensive content.The reported maximum spot-check overlap was 0.129.
  • Ethics Statement: Track-A experiments use task-disjointness, skill-disjointness, and decontamination probes to test separation between synthetic and real data.The synthetic-positive skills do not overlap with real-positive skills, making synth-only evaluation a cross-skill-subspace test.

B Track B Details and Quality Gates

Track B uses multi-axis judging and leakage checks to constrain synthetic supervision, while explicit diagnostics expose calibration and catalog-coverage limits. The locked build preserves held-out-skill integrity but should not be interpreted as distributionally equivalent to real data.

  • Quality Gates: Track B quality gates bound interpretation rather than establish distributional equivalence between synthetic supervision and the real harvest.The gates diagnose mismatch while allowing retained rows to remain valid.
  • Quality Gates: Track B retains rows using realism, necessity, sufficiency, and reward-plausibility judgments, with realism requiring at least 4 and other axes at least 3.Rows receive lower contrastive weight when their judged quality is marginal.
  • Quality Gates: The reward-weight distribution differs sharply between real and synthetic data, with KS D=0.31 and p<10^-80, making weighting calibration a limitation.The authors interpret this as a mismatch in the weighting signal, not evidence that retained assignments are implausible.
  • Quality Gates: Only 140 of 26,947 retrieval-eligible skills appear as positives, concentrating supervision on a narrow supported subset rather than the catalog long tail.This limits interpretation as full-pool adaptation and motivates retaining a frozen-retriever fallback.
  • Quality Gates: The second-judge audit supports a conservative, high-precision retained set because GPT-5.5 is stricter than Claude Opus 4.8.Agreement is strongest on necessity, with Pearson r=0.79.
  • Quality Gates: 0 held-out-skill violations were found across positive, negative, and seed checks, preserving the validity of held-out evaluation.A single violation would invalidate held-out-evaluation numbers.

C Lambda Sweep and Seed Variance

The forgetting-mitigation sweep identifies LoRA rank as the main retention–capacity trade-off, while conservative low-rank updates remain stable across seeds. Higher-capacity or unregularized updates are more variable and can lose real/OOD performance.

  • Lambda Sweep: LoRA rank changes Recall@10 by up to 0.050 across rings, whereas anchor-weight sweeps change it by at most 0.012.Rank therefore controls the main performance-retention trade-off in this sweep.
  • Seed Variance: Conservative r=8 runs show effectively zero variance on Rings 1 and 3 and negligible variance on the synthetic ring.The seed set contains three runs, as specified by Table 7.
  • Seed Variance: Aggressive unregularized runs are less stable, with Ring 3 variance reaching 0.115 for r=16 and 0.076 for r=32.Higher-rank conservative runs also become more seed-sensitive.
  • Overall Interpretation: Lower-capacity regularized updates retain real/OOD behavior while preserving most synthetic gains, unlike aggressive updates.The authors characterize aggressive-update regression as a stability-plasticity failure consistent with representational or ranking-function drift.

E Out of Distribution Evaluation on Large Sample Size

Large-sample BEIR evaluation reproduces the paper’s central pattern: forgetting mitigation preserves or slightly improves OOD retrieval, whereas aggressive LoRA fine-tuning degrades it. Per-task analyses support practical equivalence to the frozen baseline for regularized methods, with small real-task samples limiting certainty.

  • Per-Task Analysis: Real-task samples are small—Ring 1 has N=21 and Ring 3 has N=10—so some paired-test results remain ambiguous.The study therefore supplements aggregate results with per-task paired differences.
  • BEIR Evaluation: Across six BEIR datasets, forgetting-mitigated methods preserve or slightly improve Recall@10 and MRR@10 relative to the frozen baseline.Aggressive LoRA consistently reduces performance, often substantially.
  • BEIR Evaluation: Regularized methods outperform aggressive LoRA on both in-distribution and large OOD evaluations.This repeats the pattern observed in the three-ring experiments.

F Deployment

The deployed system combines conservative retriever and reranker fine-tuning with FAISS retrieval and score blending in a stateless MCP server. On an H100, it provides low-latency, low-memory GPU serving, while CPU-only execution is substantially slower.

  • Deployment Architecture: The MCP server exposes retrieve_skills, retrieve_skills_debug, get_skill_detail, and health through a stateless streamable HTTP interface.It runs the same three-stage pipeline used in the experiments.
  • Deployment Architecture: The deployed pipeline uses a conservative-anchor Qwen3-Embedding-0.6B retriever, FAISS exact search over 34,396 skills, and an LwF-trained Qwen3-Reranker-0.6B.Retriever and reranker scores are combined using min-max normalization and α=0.55.
  • Performance: On one NVIDIA H100, the full query path takes 169 ms p50 and 175 ms p95, sustains 5.9 queries/s, and uses 4.3 GB of GPU memory.The reported memory footprint fits the system on GPUs with roughly 8–12 GB of VRAM.
  • Performance: CPU-only serving reaches 8.48 s p50 and 12.89 s p95 latency with 0.11 QPS sequential throughput.The CPU mode uses approximately 6.1 GiB resident memory.
Loading 2609.10750v1…