PaperlayerView exact revision on arXiv

Source-linked AI summary

SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models

Anke Tang, Li Shen, Yong Luo, Shuai Xie, Han Hu, Lefei Zhang, Bo Du, Dacheng Tao

arXiv:2408.10174v2cs.LGcs.AI

TL;DR

Deep model fusion can reduce the need for extensive retraining, but parameter interference and limited interpretability remain challenges. The paper analyzes fine-tuning and interference through subspaces and proposes zero-shot SMILE construction. Across reported settings, fusion retains much of the individual-model performance with comparatively small parameter increases.

  • Problem

    Model fusion faces parameter interference and limited interpretability, while training large models on extensive datasets is increasingly costly.

    Source: §1 · pp. 1–2

  • Method

    SMILE constructs a sparse mixture-of-experts model from source models without extra data or further training, using subspace analysis to study fine-tuning and parameter interference.

    Source: §1 · p. 2

  • Results

    Across vision and language experiments, full-fine-tuned fusion reaches around 98–99% of eight individual models’ performance with about 50% more parameters, while LoRA retains 99% with a 2% increase.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that fine-tuning largely preserves important pretrained weights while adapting through less-significant or unused dimensions, and that dimensional expansion makes interference more manageable.

    Source: §1 · p. 2

    The interference formulation is scoped to linear layers with a common input across task-specific models.

    Source: §3 · p. 5

Abstract

from arXiv · show

Deep model training on extensive datasets is increasingly becoming cost-prohibitive, prompting the widespread adoption of deep model fusion techniques to leverage knowledge from pre-existing models. From simple weight averaging to more sophisticated methods like AdaMerging, model fusion effectively improves model performance and accelerates the development of new models. However, potential interference between parameters of individual models and the lack of interpretability in the fusion progress remain significant challenges. Existing methods often try to resolve the parameter interference issue by evaluating attributes of parameters, such as their magnitude or sign, or by parameter pruning. In this study, we begin by examining the fine-tuning of linear layers through the lens of subspace analysis and explicitly define parameter interference as an optimization problem to shed light on this subject. Subsequently, we introduce an innovative approach to model fusion called zero-shot Sparse MIxture of Low-rank Experts (SMILE) construction, which allows for the upscaling of source models into an MoE model without extra data or further training. Our approach relies on the observation that fine-tuning mostly keeps the important parts from the pre-training, but it uses less significant or unused areas to adapt to new tasks. Also, the issue of parameter interference, which is intrinsically intractable in the original parameter space, can be managed by expanding the dimensions. We conduct extensive experiments across diverse scenarios, such as image classification and text generation tasks, using full fine-tuning and LoRA fine-tuning, and we apply our method to large language models (CLIP models, Flan-T5 models, and Mistral-7B models), highlighting the adaptability and scalability of SMILE. Code is available at https://github.com/tanganke/fusion_bench

1 Introduction

Model fusion combines pretrained models to reduce the costs of training from scratch, but parameter interference and limited interpretability remain challenges. The paper analyzes fine-tuning and interference through subspaces, introduces zero-shot SMILE, and reports strong performance with modest parameter increases.

1 Introduction

  • Deep model fusion can combine knowledge from existing models without extensive retraining, but interference and limited interpretability remain challenges.The paper highlights these issues especially for complex, high-dimensional, nonlinear architectures.Source: §1 · pp. 1–2
  • The paper examines linear-layer fine-tuning through subspace analysis and formulates parameter interference as an optimization problem.The analysis decomposes fine-tuned predictions into pretrained and task-specific components.Source: §1 · p. 2
  • SMILE constructs a more versatile mixture-of-experts model from existing source models without additional data or training.The authors describe zero-shot construction as reducing the time and resources typically needed for adaptation.Source: §1 · p. 2
  • Fine-tuning largely preserves important pretrained weights while using less-significant or previously unused dimensions for task adaptation.The authors report that expanding dimensionality makes parameter interference more manageable.Source: §1 · p. 2
  • With about 50% more parameters, full-fine-tuned fusion reaches around 98–99% of eight individual models’ performance; LoRA fusion retains 99% with a 2% increase.Experiments span vision and language tasks and models.Source: §1 · p. 2

2 Rethinking Model Fine-Tuning From a Subspace Perspective

This section uses singular-value subspaces to analyze how fine-tuning changes linear-layer weights. Its experiments indicate that updates in less-important or unused dimensions recover more downstream performance than updates restricted to the most important pretrained subspace.

2 Rethinking Model Fine-Tuning From a Subspace Perspective

  • The section decomposes pretrained linear-layer weights using reduced SVD, with singular values ordered from largest to smallest.The matrices of left and right singular vectors and the diagonal singular-value matrix form the decomposition.Source: §2 · p. 3
  • For a fine-tuned linear layer, the output separates into a pretrained contribution and an update contribution whose coefficients measure update alignment with singular-vector directions.The update coefficients also indicate which input-output transformations are enhanced or suppressed during fine-tuning.Source: §2 · p. 4
  • The authors partition update subspaces by singular-value importance: zone I contains values accounting for the top 50% of their cumulative sum and is associated with preserving pretrained information.The remaining cumulative half defines zone II, while zone III lies beyond the pretrained weight matrix’s rank.Source: §2 · p. 4
  • Updates projected onto subspaces II & III produce significant downstream gains, nearly reaching fine-tuned performance, while subspace I yields slight or no improvement.Subspace II gives moderate improvement; the comparisons use CLIP-ViT-B/32 fine-tuned on eight downstream tasks.Source: §2 · pp. 4–5
  • Fine-tuning largely preserves important pretrained features while using less-significant or previously unused weight-space dimensions for task-specific learning.The authors infer this from the subspace projection comparisons.Source: §2 · p. 5

3 Parameter Interference Between Task-Specific Models

This section frames interference among task-specific fine-tuned models as a problem of matching merged-model outputs to each task’s fine-tuned outputs. Its analysis links interference to activated input dimensions and identifies additional orthogonal dimensions as a way to make it more manageable.

3 Parameter Interference Between Task-Specific Models

  • The analysis formulates merging as minimizing the discrepancy between the merged model’s output and each task-specific fine-tuned model’s output on that task’s input.It assumes linear layers, a common input form, and a shared pretrained contribution across tasks.Source: §3 · p. 5
  • Weight interference does not occur on subspaces orthogonal to the input, so enlarging the input dimension can increase room for updates to coexist.The authors connect this to stronger performance from merging larger models with more dimensional redundancy.Source: §3 · p. 5
  • When an input activates a dimension, interference is inevitable unless task-domain gaps make the activation dimensions disjoint.The authors also state that bias-term solutions vary by task and suggest fixing the bias during fine-tuning.Source: §3 · p. 5

4 Resolving Parameter Interference using Sparse Mixture of Low-Rank Experts

SMILE upscales source models into a zero-shot mixture of experts with a shared pre-trained component, a router, and low-rank task experts. It uses truncated updates and input-dependent routing to combine task-specific components without data or further training.

4 Resolving Parameter Interference using Sparse Mixture of Low-Rank Experts

  • The upscaled model combines a shared pre-trained part, router, and low-rank experts, and operates without data or further training.The paper also describes this construction as a zero-shot SMILE model.Source: §4 · p. 6
  • SMILE expresses a merged model as the pre-trained output plus input-selected, weighted task updates, with the selection intended to minimize merging error.The ideal task weights form a one-hot distribution, while a naive implementation would require router training and storage for all fine-tuned weights.Source: §4 · p. 6
  • SMILE applies SVD to each task update and retains its top k directions, forming a rank-k approximation that minimizes Frobenius reconstruction error.The omitted terms correspond to less significant dimensions.Source: §4 · pp. 6–7
  • The router uses L2 norms of input projections onto low-rank matrices as logits; excessively large k_gate can reduce routing distinctiveness.The authors report k_gate values of 4 or 8 as good choices for most tasks.Source: §4 · p. 7
  • With T=8, k_gate=4, k=32, K=1, and m=n=1024, SMILE adds 565K parameters (53.9% of the original count) and activates 99K per token (9.4%).The activated-parameter count is for each input token.Source: §4 · p. 7
  • SMILE extends to LoRA fine-tuned models, with k_gate set below the LoRA rank r_LoRA and k no greater than r_LoRA.The update is decomposed using SVD.Source: §4 · p. 7

5 Experiments

Experiments evaluate SMILE across CLIP image classification, Flan-T5 text generation, and Mistral-7B benchmarks, alongside hyperparameter and routing analyses. The reported results show competitive multi-task performance with relatively small parameter increases, while expert-rank and routing choices shape performance and cost.

5.1 Multi-Task Model Fusion on Open-Vocabulary Image Classification Tasks

  • 99.3% of individual model performance at 2.56 times one-model parameters is reported for CLIP-ViT-L/14 SMILE, versus 6.40 times for Weight-Ensembling MoE.This setting uses kgate = 16 and k = 128 across eight image-classification tasks.Source: Table 3 · p. 8, §5.1 · p. 8
  • The CLIP evaluation uses eight open-vocabulary image-classification tasks, freezes each pre-trained model's text encoder, and fine-tunes only its vision encoder.SMILE is described as requiring neither additional labeled samples nor test-time adaptation.Source: §5.1 · p. 8
  • Across CLIP-ViT-L/14 eight-task fusion with kgate = 16, Figure 4 varies k from 4 to 128 to show the performance–model-size trade-off.The passage describes the figures as demonstrating SMILE's effectiveness and the trade-off between performance and model size.Source: Figure 4 · p. 8, §5.1 · p. 8

5.2 Multi-Task Model Fusion on Text Generation Tasks

  • 99.0% of individual performance at 1.52 times single-model parameters is reported for full fine-tuning, while LoRA reaches 99.3% at 1.02 times.The results cover Flan-T5-base models fine-tuned on eight GLUE tasks; SMILE is reported to outperform other fusion methods in full fine-tuning and surpass them in LoRA.Source: Table 4 · p. 9, Table 5 · p. 9, §5.2 · p. 9

5.3 Spare Mixture of Low-Rank Experts Analysis

  • Increasing k and kgate generally improves performance with diminishing returns as parameter count grows, while relatively small values can achieve near-optimal performance.The analysis covers Flan-T5-Base and CLIP-ViT models under full and LoRA fine-tuning.Source: §5.3 · p. 9, Figure 5 · p. 10
  • Average accuracy decreases slightly as Top-K routing K increases, while individual-task accuracy remains relatively stable across K values.The routing analysis uses CLIP-ViT-B/32 across eight image-classification tasks; the authors note that using more experts per input is not necessary in this setting.Source: §5.3 · pp. 9–10, Figure 6 · p. 10

5.4 Scalability to Large-Scale Models (Mistral-7B models)

  • On Mistral-7B GSM8K, scores generally improve as expert rank k increases, with stronger gains for M0;1 and M0;123 at fixed kgate = 8.The authors interpret this pattern as evidence that routers select suitable experts for math problems.Source: Figure 7 · p. 10, §5.4 · p. 10
  • The Mistral-7B comparison reports specialized strengths among individual experts, with M1 strongest on GSM8K and M3 on ARC Challenge.The paper states that combining expert knowledge enhances overall performance across a diverse set of tasks.Source: Table 6 · p. 11, §5.4 · p. 11

6 Related Work

The related work surveys model fusion through mode connectivity, alignment, and model mixing, alongside the development of mixture-of-experts routing and expert design.

6 Related Work

  • Mixture-of-experts research has focused substantially on routing and expert design, including top-one token routing and linear assignment for token-expert affinities.The section also describes building LoRA adapter libraries and selecting adapters by input without retraining.Source: §6 · p. 11
  • Mode connectivity motivates weight interpolation for fusion, while diverse model structures create challenges that alignment methods address through matching or permutation-based techniques.The cited approaches include activation or weight matching, channel-wise graph matching, and permutation invariance.Source: §6 · p. 11
  • Model mixing combines models through gating or depth concatenation, while MoEerging is identified as a related model-merging line of work.The discussion points readers to reviews of MoE and MoEerging.Source: §6 · p. 11

7 Conclusion, Limitations, and Future Work

The conclusion presents SMILE as a zero-shot model-fusion approach that avoids additional training data or processes. It also notes extra computational overhead from sparse activation and identifies several directions for future work.

7 Conclusion, Limitations, and Future Work

  • SMILE is presented as a zero-shot model-fusion method that requires no additional training data or processes, although sparse activation still adds computational overhead that grows with experts.The conclusion characterizes the approach as practical while noting this overhead.Source: §7 · p. 11
  • Future work could identify task-relevant subspaces for more targeted fine-tuning and examine SMILE for multimodal large language models.The proposed multimodal direction treats different data modalities as separate experts.Source: §7 · p. 11
  • The authors also propose exploring multi-objective routing weights and dynamically adjusting the number of experts K per token to improve efficiency without sacrificing performance.Source: §7 · p. 11

A Projecting Fine-tuned Updates onto Subspaces

This section uses SVD-based subspaces to examine task-specific updates relative to pre-trained linear-layer weights, then evaluates projected weights on downstream tasks.

A Projecting Fine-tuned Updates onto Subspaces

  • SVD of the pre-trained weight matrix provides a basis for examining task-specific information in updates after downstream fine-tuning.The experiments analyze the distribution of task-specific information across subspaces of the weight matrix.Source: A Projecting Fine-tuned Updates onto… · p. 15
  • The analysis partitions singular directions into Zone I, the dominant directions; Zone II, the remaining nonzero directions; and Zone III, directions beyond the pre-trained weight matrix’s rank.Zone I is selected so its singular values account for approximately the full sum; Zone II spans rhalf + 1 through rank r, while Zone III includes indices beyond r.Source: A Projecting Fine-tuned Updates onto… · p. 15
  • Downstream accuracy is evaluated by substituting modified weight matrices for the original pre-trained weights while keeping other layers unchanged.The modifications are projections onto the analyzed subspaces.Source: A Projecting Fine-tuned Updates onto… · p. 15

B The Gradient Flow During Fine-tuning

The section analyzes fine-tuning of linear layers through gradient flow and an NTK perspective, then uses update directions to motivate SMILE’s routing logits.

B The Gradient Flow During Fine-tuning

  • The network is decomposed into pre-linear, linear, and post-linear components, with the analysis focusing on the linear layer.The linear layer is represented using the fine-tuned weights and bias.Source: B The Gradient Flow During Fine-tuning · pp. 15–16
  • SGD updates the linear-layer weights and bias using loss gradients, and mini-batch SGD represents the gradient update as an expectation over samples.The expected update is considered when the fine-tuned weights are close to the pre-trained weights.Source: B The Gradient Flow During Fine-tuning · p. 16
  • SMILE’s routing logits project inputs onto the top kgate right singular vectors of each task’s weight update, which represent directions important to task-specific updates.The choice is motivated by the expected similarity between those vectors and the right singular vectors of the expected update.Source: B The Gradient Flow During Fine-tuning · p. 16
  • The NTK perspective treats the linear-layer kernel as a constant matrix for a given input.The kernel is defined from gradients with respect to the pre-trained linear-layer parameters.Source: B The Gradient Flow During Fine-tuning · p. 16

C Fine-Tuned Model Performance

The reported baselines show that fine-tuning improves target-task performance, while transfer across tasks can be negative or beneficial; performance also varies with task difficulty and model size.

C Fine-Tuned Model Performance

  • Fine-tuning consistently improves performance on the target task compared with the pre-trained model across the reported model types.These results serve as baselines for evaluating the fusion method.Source: C Fine-Tuned Model Performance · p. 18
  • The MNLI-tuned model performs well on QNLI, suggesting transfer of relevant knowledge between these natural-language-inference tasks.The results are for Flan-T5-Base models on GLUE tasks.Source: C Fine-Tuned Model Performance · p. 18
  • EuroSAT and MNIST are described as easier CLIP-ViT tasks, while SUN397, Cars, and DTD are more challenging.This pattern is observed in the diagonal entries for the CLIP-ViT models.Source: C Fine-Tuned Model Performance · p. 18
  • CLIP-ViT-L/14 generally performs better than CLIP-ViT-B/32, indicating that model capacity relates to task-specific performance and generalization.Source: C Fine-Tuned Model Performance · p. 18

D Hyperparameter Analysis

The analysis finds trade-offs between SMILE’s performance and parameter overhead as hyperparameters vary. Recommended settings differ across CLIP and Flan-T5 configurations.

D Hyperparameter Analysis

  • Increasing k generally improves CLIP performance while increasing model complexity, whereas performance starts to decrease when kgate becomes too large.The authors attribute the latter trend to a potentially less discriminative gating mechanism.Source: D Hyperparameter Analysis · p. 18, Figure 8 · p. 19
  • Around 98% of fine-tuned-model performance is achieved with 20% of the total parameters needed to maintain eight individual fine-tuned models.The recommended CLIP settings are k ≈32 and kgate ≈16 for ViT-B/32, and k ≈64 and kgate ≈8 for ViT-L/14.Source: D Hyperparameter Analysis · p. 18, Figure 8 · p. 19
  • About 90% of individual-model average multi-task performance is achieved with 7% parameter overhead.Source: D Hyperparameter Analysis · p. 19
  • Flan-T5-Base performance is relatively stable across configurations: 85.1–85.6 for full fine-tuning and 83.8–84.0 for LoRA, with LoRA adding 2%–7% of the original dense model’s parameters.The suggested settings are k ≈32 and kgate ≈8 for full fine-tuning, and k ≈8 and kgate ≈2 for LoRA.Source: D Hyperparameter Analysis · p. 19

E Large-Scale Model Experiments

The large-scale experiments examine SMILE on CLIP and Mistral-7B models, including performance and model-size trade-offs as hyperparameters vary. The Mistral-7B evaluation uses three expert models and full-fine-tuned experts, leaving larger expert pools and LoRA experts for future exploration.

E Large-Scale Model Experiments

  • Figure 9 analyzes how k and kgate affect average performance and normalized parameter count for CLIP-ViT-L/14 across eight image-classification datasets.It also shows average accuracy for pre-trained and individual fine-tuned models.Source: Figure 9 · p. 20
  • The Mistral-7B evaluation combines the Mistralai pre-trained model with MetaMath, Dolphin, and Speechless-Code experts, testing k values from 8 to 512 with kgate fixed at 8.Evaluation covers MMLU, TruthfulQA, GSM8K, and ARC Challenge.Source: E Large-Scale Model Experiments · pp. 19–20
  • With k = 8, the 7.3B-parameter Mistral-7B SMILE model matches or exceeds the pre-trained model on all four tasks; at k = 512 and 11.2B parameters, it approaches individual experts.The tasks are MMLU, TruthfulQA, GSM8K, and ARC Challenge; Table 11 compares individual models and SMILE models across k values, with kgate fixed at 8.Source: E Large-Scale Model Experiments · p. 20, Table 11 · p. 21
  • Mistral-7B experiments use three experts, compared with eight for CLIP and Flan-T5, so they may not fully demonstrate SMILE’s capabilities with larger, more diverse pools.The authors identify adding expert models as a direction for future work.Source: E Large-Scale Model Experiments · p. 20
  • The Mistral-7B experiments use fully fine-tuned experts; the authors suggest LoRA-fine-tuned experts could better exploit SMILE’s potential.The authors note that LoRA fine-tuning reduced the additional parameters needed for comparable performance in Flan-T5 experiments.Source: E Large-Scale Model Experiments · p. 21
Loading 2408.10174v2…