Can One Adapted Model Do It All? Fine-Tuning Strategy Selection for Customer Support LLMs

• Specialists bring task-specific strength but require more storage, endpoints, and reliable routing; LoRA and sequential updates offer different cost and retention trade-offs.LoRA can reduce storage and update costs but may not match full fine-tuning, while sequential updates can degrade earlier tasks.Source: §1 · p. 1
• The study compares five adaptation strategies under a controlled setup because prior methods were rarely compared on the same models, data, and evaluation protocol.The comparison spans thirteen models and more than 200 trained models.Source: §1 · pp. 1–2
• Multi-task full fine-tuning is the strongest default across the thirteen models.Source: §1 · p. 2
• Specialists often perform well on their target tasks but degrade sharply elsewhere, so imperfect routing matters.Source: §1 · p. 2
• Sequential full fine-tuning substantially harms earlier skills while LoRA can mitigate forgetting; merging a specialist with its base improves off-task retention with small target-task loss.Source: §1 · p. 2
• The authors offer strategy-selection recommendations based on task similarity, model size, and update frequency.Source: §1 · p. 2

• Full fine-tuning updates all model parameters, whereas LoRA updates a small parameter set while keeping the base model frozen; multi-task and task-specific tuning bring different generalization and specialization trade-offs.Source: §2 · p. 2
• Sequential adaptation supports adding tasks over time but risks catastrophic forgetting, in which later updates degrade earlier capabilities.Source: §2 · p. 2
• Post-hoc merging combines independently fine-tuned parameters without retraining, and the paper investigates whether it preserves customer-support skills; these strategies had not been systematically compared under a unified protocol.The text distinguishes this from adapter-combination methods that use supervision or few-shot examples.Source: §2 · p. 2

3.1 Adaptation Strategies

• Single-task models are evaluated on their target and remaining tasks, while multi-task training uses one checkpoint intended to serve all tasks in an evaluation suite.The study also compares full fine-tuning with LoRA and evaluates final sequential-update checkpoints across tasks to measure forgetting.Source: §3.1 · p. 2

3.2.1 Public Academic Suite

• The public academic suite contains four tasks and 56,752 training and 6,627 evaluation samples, including intent classification, summarization, retrieval-grounded QA, and tool-use decisions.WixQA uses simulated training data and expert-written evaluation data, which differ in document length, formatting, and citation style.Source: §3.2.1 · p. 3

3.2.2 Proprietary Internal Suite

• The proprietary internal suite contains four tasks on real-world support transcripts, with 17,791 training and 2,086 evaluation examples.Tasks cover action items, meeting summaries, call-purpose classification, and call outcomes.Source: §3.2.2 · p. 3

3.3 Models

• The model set spans thirteen models from five families and 0.6B–32B parameters, including matched 4B Qwen3 and Qwen3.5 variants.Source: §3.3 · p. 3

4.1 Main Results

• ST-Full leads on the public suite by 1.2 points on average, while internal-suite gaps stay within ±2.5 points; MT-LoRA trails MT-Full on three internal models.Table 1 compares zero-shot, single-task, and multi-task adaptation across full fine-tuning and LoRA.Source: §4.1 · p. 4, Table 1 · p. 4

4.2 Cross-Task Robustness

• Qwen3-8B’s chatbot-response specialist scores 42.3 in-task but 18.1 on QA and 14.4 on dialogue summarization; internal specialists also show off-task degradation.The call-outcome specialist scores below 5.0 across generation tasks; findings suggest output-format similarity may predict transfer better than topic similarity.Source: §4.2 · pp. 4–5, Figure 2 · p. 5

4.3 Sequential Fine-Tuning

• Sequential full fine-tuning forgets earlier skills, while sequential LoRA retains intent classification at 78.6 and action items at 57.8 in the tested order.Reversing the order again leaves earlier tasks most affected; the authors advise training the most critical task last.Source: §4.3 · p. 5, Table 2 · p. 5

4.4 Model Merging

• Merging a specialist with its base improves off-task robustness; for models ≥4B, same-task loss is at most 2.6 points, while Qwen3.5-9B improves both.Increasing α favors specialization over robustness; merging multiple specialists does not match multi-task fine-tuning or improve average score as specialists are added.Source: §4.4 · p. 5, Table 3 · p. 5, Table 4 · p. 6

4.5 Beyond Lexical Overlap

• Across generation metrics, fine-tuned strategies outperform zero-shot, and ST-Full and MT-Full remain comparable.LLM-judge differences are very minor; BERTScore favors MT-Full on the public suite and marginally favors ST-Full on the internal suite.Source: §4.5 · p. 6, Table 5 · p. 6

4.6 Deployment Considerations

• The deployment choice depends on endpoint limits, update frequency, and routing reliability; MT-Full is preferred when all task data are available.When multi-task training is infeasible, sequential LoRA supports updates with less forgetting, while merging specialists with the base improves off-task robustness.Source: §4.6 · p. 6

5 Conclusion

• Across thirteen customer-support LLMs, strategies differ beyond same-task performance: specialists can fail off-task, sequential full fine-tuning forgets earlier skills, and LoRA preserves them better.Merging a specialist with its base improves robustness, while merging multiple specialists does not reliably replace joint training.Source: §5 · p. 6
• Single-task and multi-task full fine-tuning perform similarly for most models, with no consistent model-size or capability threshold determining which is better.Source: §5 · p. 6
• The conclusion recommends evaluating cross-task behavior and skill retention rather than relying on model scale or in-task performance alone.Source: §5 · p. 6

Model coverage

• The full strategy grid covers seven of thirteen models; all tested bases are instruction-tuned, and models above 32B or non-instruct checkpoints are untested.Source: Limitations · p. 7

Single language

• All datasets are English-only, so whether the reported task-specialist, multi-task, and sequential-update patterns hold for multilingual support conversations remains untested.Source: Limitations · p. 7

α choice

• Main-text merge-with-base tables fix α = 0.5 across 32 configurations, although α ∈ {0.3, 0.7} operating points and a full sweep are also reported.Source: Limitations · p. 7

Metric scope

• ROUGE-1 is the deterministic generation metric for identical strategy comparisons, while BERTScore and an LLM judge provide additional generation-task evaluations.Source: Limitations · p. 7

Reproducibility

• Public-data runs are independently reproducible with released configurations, prompts, preprocessing, and code, but proprietary internal datasets cannot be released for independent verification.Source: Limitations · p. 7

Broader Impact

• Choosing an appropriate strategy can reduce wasted training compute and its associated energy footprint.Source: Broader Impact and Ethics Statement · p. 7
• The merge-with-base finding suggests single-task fine-tunings can be made cross-task robust without additional training, yielding direct compute savings.Source: Broader Impact and Ethics Statement · p. 7

Ethics Statement

• The internal suite uses PII-redacted enterprise call transcripts, and the proprietary models and training data are not released.Source: Broader Impact and Ethics Statement · p. 7

A Task Prompts

• Task prompts use square-bracketed placeholders and braces for task-specific content filled in at runtime.Source: A Task Prompts · p. 9

A.1 Public Suite

• The public intent-classification prompt requests JSON fields for intent, domain, language, and source dataset.Source: §A.1 · p. 9
• In the zero-shot condition, intent and domain use fixed label sets, while language is specified as an ISO 639-1 code.Source: §A.1 · p. 9
• Public-suite templates also specify formats for retrieved-content QA, dialogue summaries, and tool-use decisions.Source: §A.1 · pp. 9–10

A.2 Internal Suite

• Internal prompts place a task-specific instruction and call transcript in named fields, with the transcript bracketed by start and end markers.Source: §A.2 · p. 10
• Internal tasks include generating unfinished follow-up TODOs and selecting a call-outcome category from a provided list; call outcome uses 30 fixed categories.Source: §A.2 · p. 10
• Internal meeting-summary prompts vary desired length and format, while purpose-of-call prompts supply an example-specific shortlist and are excluded from output-format-validity analysis.Source: §A.2 · p. 10

B Full Model Coverage

• Table 6 reports public- and internal-suite averages for all thirteen bases alongside each base’s zero-shot competence.Source: B Full Model Coverage · p. 10

C Training and Inference Parameters

• Training used AdamW with a 1.5 × 10−5 learning rate, cosine scheduling with 10% warmup, and a maximum of five epochs.Runs used per-device batch size 1, gradient checkpointing without accumulation, and a maximum context length of 16K tokens.Source: C Training and Inference Parameters · p. 10
• LoRA used rank 16, α = 32, dropout 0.05, and a 1 × 10−4 learning rate.Source: C Training and Inference Parameters · p. 10
• Experiments ran across 56 H200 GPUs using LLaMA-Factory and DeepSpeed ZeRO-2 for models up to 8B or ZeRO-3 for 9B; inference used vLLM with tensor-parallel-8 on eight A100-80GB GPUs.Inference temperature was 0.Source: C Training and Inference Parameters · p. 10

D Model Merging: Effect of the Interpolation Weight α

• The sweep interpolates between the base model at α = 0 and the unmerged specialist at α = 1, evaluating the merged model on all four internal tasks.The merged weights are defined as θmerged = α θFT + (1 − α) θbase.Source: D Model Merging: Effect of the… · p. 10
• α ∈ [0.15, 0.30] gives the reported best trade-off: in-task quality is within a few points of its maximum while off-task skills are largely retained.The main-text tables use α = 0.5 for consistency, while the authors recommend choosing other values according to in-task versus off-task priorities.Source: D Model Merging: Effect of the… · p. 11
• The call-outcome score reaches 77.8 by α = 0.2 and 82.2 by α = 0.3, compared with 84.4 for the unmerged specialist.The in-task score rises sharply and saturates early, so essentially all in-task benefit is reached by about α = 0.3.Source: D Model Merging: Effect of the… · p. 11
• As α increases, off-task action-items scores fall from 46.7 at α = 0.2 to 3.6 for the pure specialist, and meeting-summaries scores fall from 53.8 to 3.5.Purpose of call is the most fragile off-task skill, losing most capability between α = 0.2 and α = 0.4.Source: D Model Merging: Effect of the… · p. 11

E Training and Serving Cost

• MT-Full processes the same total data as four ST-Full runs, so it does not reduce token-level training compute, but needs one job, checkpoint, endpoint, and no router.Source: E Training and Serving Cost · p. 11
• At a fixed model size, adaptation strategies have similar per-request inference costs, while deploying N specialists multiplies serving need by N.Source: E Training and Serving Cost · p. 11
• Table 7 reports measured wall-clock time for one five-epoch MT-Full run over four tasks using eight H200 GPUs.Source: Table 7 · p. 11, E Training and Serving Cost · p. 11