Source-linked AI summary
Upcycling Language Models for Multiple Specialized Domains
Mingzi Cao, Xi Wang, Nikolaos Aletras
TL;DR
Multi-domain adaptation must prioritize relevant domain knowledge while limiting interference, but LoRA-based adapters can select irrelevant domains and remain constrained by low-rank structure. DS-Upcycling adds full MLP experts with domain-aware router supervision and updates only the corresponding expert and router during adaptation. It achieves the highest average performance across specialized and general domains, with 90–98% specialized-domain routing accuracy after adaptation.
Problem
Existing multi-domain LoRA approaches either weight adapters equally or activate all adapters, including irrelevant ones, while their low-rank structure constrains performance.
Source: §1 · p. 1Method
DS-Upcycling adds full MLP experts and uses domain-aware router supervision, then adapts each domain by updating only its corresponding expert and the router.
Source: §1 · p. 2Results
Takeaways & Limitations
Domain-aware routing and domain-specific expert updates provide a multi-domain adaptation approach that selectively activates relevant expertise while retaining general performance.
Source: §1 · p. 2, §7 · p. 8The static parameter footprint stored in GPU memory scales linearly with the number of instantiated domains, despite bounded inference computation and latency from sparse Top-2 activation.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Effective multi-domain adaptation of large language models requires prioritizing relevant domain knowledge while minimizing interference from irrelevant domains. Existing approaches commonly use Low-Rank Adaptation (LoRA) to train a separate adapter for each domain and combine these adapters within one model. However, they either weight adapters equally regardless of the input domain or dynamically weight adapters while still activating all of them including irrelevant ones. Additionally, while efficient, their low-rank structure limits performance. In contrast, upcycling is a model expansion method that adds experts and a router to a dense Transformer layer offering larger capacity than LoRA approaches. In this paper, we propose Domain-Specialized Upcycling (DS-Upcycling) for multi-domain adaptation. We introduce a new domain-aware supervision that allows the router to effectively learn to distinguish domains by selecting the appropriate experts. During fine-tuning on each new domain, we update only the corresponding expert and router while freezing all other parameters. Experiments show that our method achieves the highest average performance across the adapted domains compared to baselines while retaining general performance. With just 256 examples per domain for router supervision, DS-Upcycling achieves 90-98% routing accuracy after adaptation compared with 43-70% for routers trained without our approach.
1 Introduction
Multi-domain adaptation seeks to integrate specialized knowledge while avoiding forgetting and cross-domain interference, but LoRA adapters may weight irrelevant domains or remain limited by low-rank structure. DS-Upcycling uses full MLP experts and domain-aware routing; it outperforms baselines across specialized and general domains, with strong routing accuracy after adaptation.
1 Introduction
- LoRA-based multi-domain methods either weight adapters equally across inputs or activate all adapters, including irrelevant ones, and their low-rank structure constrains adaptation performance.Multi-domain adaptation also faces catastrophic forgetting of general-domain capabilities and cross-domain interference.Source: §1 · p. 1
- Upcycling adds full MLP experts and a router to a dense Transformer layer, enabling selective expert activation without low-rank constraints.This approach remains underexplored for multi-domain adaptation, while recent multilingual methods rely on language-specific router signals.Source: §1 · p. 2
- DS-Upcycling trains the router with a domain-aware objective, then updates only the corresponding expert and router during adaptation while freezing other parameters.The supervision encourages general inputs to use the original MLP and specialized inputs to avoid irrelevant experts.Source: §1 · p. 2
- DS-Upcycling outperforms existing baselines across specialized and general domains in two model families.The paper also reports the highest average performance across specialized and general domains.Source: §1 · p. 2
- 90–98% routing accuracy on specialized domains follows adaptation, compared with 43–70% for routers trained without domain-aware supervision.Router supervision uses just 256 examples per domain.Source: §1 · p. 2
2 Related Work
Prior multi-domain adaptation methods face trade-offs in adapter selection, expressivity, and interference. Upcycling work has largely focused on pre-training or multilingual adaptation, whereas DS-Upcycling targets arbitrary domains with domain-aware routing and expert-specific updates.
2.1 Multi-domain Adaptation
- LoRA approaches use per-domain adapters but either weight them equally or activate all adapters, while their low-rank structure limits expressivity; shared added layers can also remain vulnerable to cross-domain interference.LLaMA-Pro inserts and trains additional transformer layers while freezing pre-trained parameters.Source: §2.1 · p. 2
2.2 Model Upcycling
- Pre-training upcycling methods use large-scale data and differing expert initialization, but distinct initialization alone does not ensure domain specialization.DS-Upcycling instead guides expert selection with domain-aware router supervision and promotes specialization through domain-specific updates.Source: §2.2 · p. 2
- Multilingual upcycling guides routing with language signals, but those signals do not transfer to specialized domains within one language.Legal and biomedical English share script and much vocabulary, offering no comparable surface-level domain signal.Source: §2.2 · p. 3
3 Domain-Specialized Upcycling
DS-Upcycling expands dense Transformer layers with domain-specific experts and trains routers to direct inputs to relevant experts. Its training stages specialize experts by domain while retaining the original dense MLP as the general expert.
3 Domain-Specialized Upcycling
- DS-Upcycling combines domain-specific representation extraction, domain-aware router supervision, and domain-specific expert updates for multi-domain adaptation.The framework is designed to adapt a single upcycled model across specialized domains.Source: §3 · p. 3
3.1 Preliminaries
- Upcycling retains the original MLP and adds experts and a router; Top-k routing selects experts whose weighted outputs form the sparse MoE output.The router retains the k largest logits and sets the others to −∞.Source: §3.1 · p. 3
3.2 Representation Extraction
- Examples from each specialized and general domain pass through the dense model; their pre-MLP hidden states are paired with domain labels for router training.The method extracts representations from a small sampled set of examples per domain.Source: §3.2 · pp. 3–4
3.3 Domain-aware Router Supervision
- Each upcycled layer gets a separate router trained with a domain-aware objective that favors the general expert for general inputs and suppresses irrelevant experts for specialized inputs.For specialized inputs, the objective still allows both the matching specialist and general expert to contribute.Source: §3.3 · p. 4
3.4 Expert Initialization
- Specialized experts copy the dense MLP, so identical initial outputs and routing weights summing to one preserve the original MLP’s output regardless of router selection.This is the stated initialization property before experts specialize during adaptation.Source: §3.4 · p. 4
3.5 Domain-specific Expert Updates
- During adaptation to domain D_d, only its expert and the router are updated; other parameters remain frozen, limiting cross-domain interference while preserving the general expert’s pre-trained knowledge.Router training on extracted hidden states is the first of the two training stages.Source: §3.5 · p. 4
4 Experimental Setup
Experiments use Llama-3.1-8B and Qwen-3-8B, comparing dense and upcycling baselines across legal, biomedical, and general-domain evaluation tasks. Training data and router supervision are sampled from the corresponding domains.
4.1 Base Models
- Experiments use the pre-trained Llama-3.1-8B and Qwen-3-8B models as base models.Both are evaluated as the starting models for adaptation.Source: §4.1 · p. 4
4.2 Baselines
- Baselines include dense fine-tuning, LoRA-based adapters, and LLaMA-Pro, alongside upcycling variants with different expert initializations; Table 1 reports specialized-domain task scores.The upcycling baselines use randomly initialized routers and update all parameters during adaptation; Table 1 marks per-task best and second-best scores.Source: §4.2 · p. 4, Table 1 · p. 5
4.3 Training Data
- 393K examples per domain are sampled uniformly across task subsets; router supervision uses 256 examples per domain, with general examples drawn from Alpaca-GPT4.The specialized training domains are legal and biomedical.Source: §4.3 · p. 5
4.4 Evaluation Tasks
- Evaluation covers biomedical, legal, and general benchmarks, with mean and standard deviation reported across three random seeds.General benchmarks assess catastrophic forgetting; the biomedical and legal task suites include domain-specific datasets and relevant MMLU subsets.Source: §4.4 · p. 5
5 Results
DS-Upcycling achieves the strongest reported average performance in biomedical, legal, and general domains. Its domain-level gains extend across specialized tasks, while general-domain declines are limited compared with the broader degradation reported for other upcycling baselines.
5.1 Domain-specific Performance
- DS-Upcycling achieves the highest biomedical averages (Llama: 68.42; Qwen: 73.34) and legal averages (Llama: 68.96; Qwen: 72.15), significantly outperforming second-best baselines.The authors report paired t-test significance at p < 0.05 and improvements over the base model on every specialized-domain task.Source: §5.1 · p. 6
- Upcycling baselines do not improve consistently: Llama-3.1-8B averages fall below the base model in both domains, while Qwen-3-8B gains legally but declines biomedically.The reported highest Llama averages are Duplication: 66.18 →50.97 (biomedical) and Random Init.: 60.30 →56.62 (legal).Source: §5.1 · p. 6
5.2 General Performance
- DS-Upcycling has the highest general-domain averages (Llama: 68.82; Qwen: 70.08), significantly outperforming second-best baselines while limiting task declines.Only two Llama tasks decline slightly, and only RTE declines for Qwen.Source: §5.2 · p. 7
- All upcycling baselines reduce general-domain averages on both model families; restricting updates to each domain's corresponding expert helps mitigate catastrophic forgetting.Even with the general expert frozen, updating all specialized experts lowers general-domain averages on both model families.Source: §5.2 · p. 7
6 Analysis
The analysis links DS-Upcycling's adaptation results to stable domain routing, specialization of identically initialized experts, and benefits from both router supervision and domain-specific updates.
6.1 Routing Quality
- DS-Upcycling maintains high routing accuracy across general, legal, and biomedical domains after adaptation, whereas randomly initialized routers can become biased toward general inputs as specialized-domain accuracy falls.For Llama Random Init., general accuracy rises from 68.31 to 99.62 while legal falls from 62.94 to 51.00 and biomedical from 61.06 to 42.69.Source: §6.1 · p. 7, Table 3 · p. 7
6.2 Expert Specialization
- DS-Upcycling's biomedical and legal experts have low update similarity despite identical initialization, unlike Duplication's highly similar updates.Lower similarity indicates less aligned update directions and serves as an indicator of expert specialization.Source: §6.2 · p. 8, Table 4 · p. 8
6.3 Ablation Study
- Domain-aware router supervision improves average performance in five of six settings; the exception is Qwen-3-8B biomedical, decreasing from 73.46 to 73.34.The ablation compares DS-Upcycling with a variant using randomly initialized routers and otherwise unchanged components.Source: §6.3 · p. 8, Table 5 · p. 8
- Restricting each expert's updates to its corresponding domain improves average performance across all six settings.The ablation evaluates this strategy against a variant that updates all specialized experts during adaptation to each domain.Source: §6.3 · p. 8, Table 5 · p. 8
7 Conclusion
DS-Upcycling achieves the highest average performance across specialized and general domains, with domain-aware router supervision and domain-specific expert updates. After adaptation, its specialized-domain routing accuracy exceeds that of existing upcycling baselines.
7 Conclusion
- DS-Upcycling offers the highest average performance across specialized and general domains.The authors attribute its prioritization of relevant domain knowledge and reduced interference to domain-aware router supervision and domain-specific expert updates.Source: §7 · p. 8
- 90–98% routing accuracy on specialized domains after adaptation compares with 43–70% for existing upcycling baselines.Source: §7 · p. 8
- Domain-specific adaptation updates only the corresponding expert and router while freezing all other parameters.Source: §7 · p. 8
Limitations
DS-Upcycling’s full MLP experts avoid low-rank expressivity bottlenecks, but its static GPU-memory footprint scales with the number of instantiated domains. The authors identify quantization or pruning as possible future ways to reduce storage for memory-constrained serving.
Limitations
- Static parameter storage in GPU memory scales linearly with the number of instantiated domains.Source: Limitations · p. 9
- Inference computation and latency remain bounded by sparse Top-2 activation.Source: Limitations · p. 9
- Expert quantization or pruning is proposed as future work to optimize storage for memory-constrained serving.Source: Limitations · p. 9
A Implementation details
The implementation upcycles odd-numbered base-model layers into sparse MoE layers, retaining the original MLP and adding two domain experts. The appendix also specifies baseline configurations and training and evaluation settings.
A Implementation details
- Odd-numbered layers become sparse MoE layers, each retaining the original MLP and adding legal and biomedical experts with Top-2 routing.New expert weights use random initialization, random perturbation replacing 50%, or partial initialization reinitializing 50%.Source: A Implementation details · p. 12
- Baselines use similar parameter counts; LoRA ranks are 1024 and LoRAMoE ranks are 1536, while LLaMA-Pro adds seven layers per group.For LLaMA-Pro, the base model is divided into four layer groups.Source: A Implementation details · p. 12
B Hyper-parameter Details
- Training uses global batch size 64 and sequence length 4096, with cosine learning-rate schedules and a lower range for biomedical adaptation.Biomedical adaptation lowers the range to 5 × 10−6–5 × 10−7 for upcycling baselines and Simple FT; evaluation is zero-shot.Source: B Hyper-parameter Details · p. 12
C Ablation Results
The appendix identifies full ablation results for specialized and general domains.
C Ablation Results
- Tables 6 and 7 present full ablation results on specialized and general domains, respectively.Source: C Ablation Results · p. 12
D Use of Generative AI Tools
The authors acknowledge using GPT-6 Astra during manuscript preparation, limited to improving the draft’s grammar and clarity.
D Use of Generative AI Tools
- GPT-6 Astra was used during manuscript preparation to improve the original draft’s grammar and clarity.The authors state that this was the tool’s limited use.Source: D Use of Generative AI Tools · p. 12
- Table 6 presents ablation performance on specialized domains.Source: Table 6 · p. 13
- Table 7 presents ablation performance on the general domain.Source: Table 7 · p. 13