PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoAdapt: Automatic Domain Discovery Enables Low-Cost Extensibility

Josh McGiff, Salma Mekaoui, Robert Shanahan, Nikola S. Nikolov

arXiv:2610.10349v1cs.LG

TL;DR

Instruction-tuned models encounter evolving, heterogeneous domains, while full retraining on expanding datasets is costly. AutoAdapt discovers latent domains, trains independent adapters with parameter-free routing, and matches single-LoRA performance across benchmarks and judge evaluation.

  • Problem

    Instruction-tuned models are deployed in evolving domains, but incorporating new domains through full retraining on combined datasets creates a compute barrier and leaves modularity unmet.

    Source: §1 · pp. 1–2

  • Method

    AutoAdapt automatically discovers latent domains, trains an independent LoRA adapter for each, and routes instructions by parameter-free cosine similarity to domain centroids.

    Source: §1 · p. 2

  • Results

    AutoAdapt matches monolithic single-LoRA performance across domain-specific benchmarks and LLM-as-a-judge evaluation.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The findings indicate AutoAdapt is a practical, extensible alternative to full retraining in heterogeneous and evolving deployment settings.

    Source: §5 · p. 8

    The evaluation is in a low-resource deployment scenario; the impact of AutoAdapt on larger models remains future work.

    Source: §5 · p. 8

Abstract

from arXiv · show

Instruction-tuned models are deployed into environments where domains are heterogeneous and evolve, yet adding new domains or data typically requires costly retraining. We present AutoAdapt, a modular framework that incorporates new domains and data via targeted single-adapter training without modifying other adapters. The framework automatically discovers latent domains, uses them to train per-domain Low-Rank Adaptation (LoRA) adapters independently in parallel and performs parameter-free routing. Across 14 domain-specific benchmarks and GPT-4o pairwise judgements, AutoAdapt achieves parity with a LoRA adapter trained on all domains without requiring full-model retraining. We also find evidence of specialisation effect convergence across independent discovery methods. Overall, training each adapter on its own domain prevents domain interference by construction, thus enabling modular, taxonomy-free domain specialisation without aggregate performance loss or full model retraining.

1 Introduction

Instruction-tuned models face evolving deployment domains, while full retraining on combined datasets creates a compute barrier. AutoAdapt discovers latent domains and trains independently routed adapters to support targeted domain updates without retraining the full model.

  • The deployment gap is between static benchmark evaluation and evolving environments where new domains and capabilities are continually required, despite the compute barrier to full retraining.Source: §1 · p. 1
  • Heterogeneous instruction data creates conflicting gradients that limit single-model specialisation, while monolithic systems are difficult to extend and updating them risks catastrophic forgetting.Source: §1 · p. 1
  • Standard jointly trained Mixture-of-Experts systems remain coupled, so adding domains requires full-system retraining; prior fixed-domain specialisation work did not establish extensibility.Source: §1 · pp. 1–2
  • AutoAdapt discovers latent domains, trains per-domain adapters independently in parallel, and routes at inference using parameter-free cosine similarity to domain centroids.Source: §1 · p. 2

2 Related Work

Related work addresses instruction-data heterogeneity through data selection and parameter-efficient tuning, while AutoAdapt partitions the full dataset and extends parameter-free routing to instruction tuning.

2.1 Instruction Tuning and Data Heterogeneity

  • Ever-expanding instruction datasets are heterogeneous, and training on dissimilar tasks introduces conflicting gradients that can degrade individual-task performance.Source: §2.1 · p. 2
  • Data-selection methods improve efficiency but discard low-scored samples, whereas AutoAdapt assigns every instruction to exactly one domain adapter without discarding examples.Source: §2.1 · p. 2

2.2 Parameter-Efficient Fine-Tuning and Adapter Routing

  • LoRA reduces adaptation cost through low-rank weight updates, while constrained-update and routing approaches address forgetting during sequential fine-tuning.Source: §2.2 · pp. 2–3
  • Prior multi-adapter routing uses a trained selector or LLM-generated routing signals; AutoAdapt extends parameter-free routing to instruction tuning.Source: §2.2 · p. 3

3 Methodology

AutoAdapt discovers latent instruction domains, trains a separate LoRA adapter for each domain, and uses centroid-based parameter-free routing. The evaluation uses a 509,174-instruction training set and compares against a single adapter trained on all domains.

3 Methodology

  • The methodology discovers latent domains, trains a dedicated LoRA adapter per domain and discovery method, and routes inference through parameter-free centroids.Source: §3 · p. 3, §3.2 · pp. 3–4

3.1 Latent Domain Discovery

  • K-means uses MiniBatch elbow analysis and validation scores to select 17 clusters; BERTopic yields 107 topics merged into 24 groups, while MNLI classifies against the same 24-label space.Source: §3.1 · p. 3

3.2 Adapter Training and Centroid-Based Routing

  • Adapters are trained with the base model frozen and components decoupled; test instructions route to the centroid with highest cosine similarity, requiring one embedding pass and no trainable routing parameters.Source: §3.2 · pp. 3–4

3.3 Training Data and Evaluation

  • The Mistral-7B-Instruct-v0.3 model is trained on 509,174 instructions from eight public datasets and evaluated against single LoRA on domain benchmarks and blind GPT-4o pairwise judgements.Source: §3.3 · p. 4, Table 1 · p. 4

4 Results and Discussion

Across judge comparisons and 14 domain-specific benchmarks, AutoAdapt is competitive with single LoRA; its adapters can also be updated or added independently, with existing routing largely stable. The results show moderate agreement between K-means and BERTopic, while MNLI produces more distinct groupings.

LLM-as-a-Judge Evaluation

  • AutoAdapt’s domain-specific adapters achieved performance comparable with single LoRA in GPT-4o pairwise judgments, with win rates of 50.5% for K-means, 50.2% for BERTopic, and 46.9% for MNLI.Comparisons used 100 test instances per domain in a blind, reference-grounded preference setting.Source: LLM-as-a-Judge Evaluation · pp. 4–5

4.1 Cross-Method Convergence

  • K-means and BERTopic showed moderate agreement (ARI 0.42, NMI 0.55), whereas MNLI groupings had near-zero ARI with both methods (0.08 and 0.07).MNLI adapters were the only AutoAdapt method to win on RACE, MBPP, CRUXEval, and HellaSwag.Source: LLM-as-a-Judge Evaluation · p. 5

Domain-Specific Benchmarks

  • Coding adapters performed poorly against single LoRA in both benchmark and judge evaluations, and the authors identify function-based embeddings as a future avenue for grouping operationally related samples.The benchmarks include code synthesis and execution-prediction tasks that are topically similar but assess different functions.Source: Domain-Specific Benchmarks · pp. 5–6
  • K-means (−0.27pp, 95% CI [−1.36, +0.82], 𝑝=0.63) and MNLI (+0.19pp, 95% CI [−0.90, +1.28], 𝑝=0.73) were statistically indistinguishable from single LoRA across benchmarks, while BERTopic was lower (−2.18pp, 95% CI [−3.28, −1.08], 𝑝<0.001).The reported CIs and p-values use an inverse-variance weighted mean of per-benchmark deltas.Source: Domain-Specific Benchmarks · p. 5, Table 2 · p. 7

4.2 Extensibility and Modularity

  • After adding 10,000 Magicoder examples, 99.9% of test-set routing assignments remained unchanged, and the updated coding adapter won more GPT-4o comparisons than single LoRA despite 19× less training data.Nearly half of the added samples routed to Algorithms & Coding (c15).Source: §4.2 · p. 7
  • Adding a legal adapter routed 99.7% of test examples unchanged; on 200 held-out LawInstruct QA examples, it was preferred in 126 comparisons against the original system and 129 against frozen single LoRA.The new adapter was trained from 1,000 legal instructions, and existing adapters were left unchanged.Source: §4.2 · p. 7
  • Training three adapters in parallel on one A100 80GB GPU took 10h 8min, 1.7× faster than single LoRA and 2.0× faster than full-model fine-tuning.Centroid routing adds relatively minimal latency, with only the active lightweight adapter swapped per request.Source: §4.2 · p. 8

5 Conclusion

AutoAdapt offers a modular alternative to monolithic instruction tuning, matching monolithic single-LoRA performance across domain-specific benchmarks and LLM-as-a-judge. Post-deployment update simulations support domain extensibility as a structural property of the framework.

5 Conclusion

  • AutoAdapt matches monolithic single-LoRA performance across domain-specific benchmarks and LLM-as-a-judge.The paper presents automatically discovered latent-domain specialisation as a modular, practical alternative to monolithic instruction tuning.Source: §5 · p. 8
  • Post-deployment update simulations demonstrate domain extensibility as a structural property of AutoAdapt.Source: §5 · p. 8
  • Future work should extend beyond low-resource deployment and investigate AutoAdapt's impact on larger models.Source: §5 · p. 8
Loading 2610.10349v1…