PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Model Fusion: A Survey

Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, Li Shen

arXiv:2309.15698v1cs.LGcs.AI

TL;DR

Deep model fusion is studied amid single-model limitations, practical resource concerns, and a lack of comprehensive synthesis. This survey organizes methods into four categories and reviews their mechanisms, applications, and challenges. It concludes by identifying directions for future fusion strategies and adaptive methods.

  • Problem

    Comprehensive reviews of deep model fusion methods and their mechanisms are lacking, despite challenges including computational load, model heterogeneity, and slow alignment.

    Source: §1 · p. 2

  • Method

    The survey classifies deep model fusion into mode connectivity, alignment, weight average, and ensemble learning, and compares their mechanisms, trade-offs, and applications.

    Source: §1 · pp. 2, 4, §7 · p. 29

  • Results

    The survey synthesizes fusion methods across four perspectives and summarizes their differences, advantages, disadvantages, applications, and engineering prospects.

    Source: §7 · p. 29

  • Takeaways & Limitations

    The authors identify novel aggregation patterns, better initial conditions, diverse ensemble frameworks, and adaptive methods for heterogeneous models and complex scenarios as future directions.

    Source: §7 · p. 29

    Combining weight averaging with fine-tuning can add overhead and may involve checkpoint proliferation or catastrophic forgetting, especially in transfer learning.

    Source: §6.2 · p. 27

Abstract

from arXiv · show

Deep model fusion/merging is an emerging technique that merges the parameters or predictions of multiple deep learning models into a single one. It combines the abilities of different models to make up for the biases and errors of a single model to achieve better performance. However, deep model fusion on large-scale deep learning models (e.g., LLMs and foundation models) faces several challenges, including high computational cost, high-dimensional parameter space, interference between different heterogeneous models, etc. Although model fusion has attracted widespread attention due to its potential to solve complex real-world tasks, there is still a lack of complete and detailed survey research on this technique. Accordingly, in order to understand the model fusion method better and promote its development, we present a comprehensive survey to summarize the recent progress. Specifically, we categorize existing deep model fusion methods as four-fold: (1) "Mode connectivity", which connects the solutions in weight space via a path of non-increasing loss, in order to obtain better initialization for model fusion; (2) "Alignment" matches units between neural networks to create better conditions for fusion; (3) "Weight average", a classical model fusion method, averages the weights of multiple models to obtain more accurate results closer to the optimal solution; (4) "Ensemble learning" combines the outputs of diverse models, which is a foundational technique for improving the accuracy and robustness of the final model. In addition, we analyze the challenges faced by deep model fusion and propose possible research directions for model fusion in the future. Our review is helpful in deeply understanding the correlation between different model fusion methods and practical application methods, which can enlighten the research in the field of deep model fusion.

1 Introduction

The introduction motivates deep model fusion as a way to combine model capabilities while addressing limits of single models and the test-time cost of ensembles. It presents a four-part taxonomy spanning parameter fusion and output aggregation, and identifies a need for a broad synthesis of methods, applications, and challenges.

1 Introduction

  • The introduction frames fusion as a response to single-model limitations, while noting that ensembles incur storage and inference costs for multiple models.It also describes large model sizes and the possibility that trained solutions lie away from the optimal point with minimum test error.Source: §1 · pp. 1–2
  • The survey addresses a gap in comprehensive reviews of deep model fusion amid computational load, model heterogeneity, and slow combinatorial alignment.It distinguishes parameter fusion from reviews focused on a single perspective, a specific scene, or multimodal fusion.Source: §1 · p. 2
  • Mode connectivity and alignment bring independently trained, distant solutions closer, while weight averaging directly combines similar models and ensemble learning aggregates predictions.Weight averaging is described as simple and efficient but potentially biased when models differ greatly in structure or parameter count; ensembles retain multiple models at test time.Source: §1 · pp. 2–3
  • The survey covers applications including federated learning, fine-tuning, distillation, and fusion on foundation models and LLMs.It describes federated learning as central aggregation of client models and fine-tuning as adapting pretrained models to particular tasks or domains.Source: §1 · pp. 3–4
  • Its four categories are mode connectivity, alignment, weight average, and ensemble learning, covering parameter-based fusion and prediction aggregation.The first three sections examine parameter fusion, while ensemble learning combines model outputs.Source: §1 · p. 4
  • The survey compares fusion approaches’ advantages, disadvantages, mechanisms, and relationships to inform the design of future methods.It also summarizes applications and current research trends.Source: §1 · p. 4

2 Mode Connectivity

Mode connectivity links neural-network solutions through weight-space paths that avoid increasing loss, allowing models along those paths to be fused or used as starting points for further optimization. The section distinguishes linear, nonlinear, and subspace paths, each with different conditions and trade-offs.

2 Mode Connectivity

  • Mode connectivity uses low-loss weight-space paths between solutions to find models for fusion, addressing cases where independently optimized networks cannot be merged directly.A direct linear path can leave the low-loss manifold and create a high-loss barrier; models along a low-loss path can be fused.Source: §2 · pp. 4–5

2.1 Linear Mode Connectivity

  • Linear mode connectivity determines a tunnel form, such as a polygonal chain, then optimizes its parameters to minimize expected loss along the path.Garipov et al. [66] use a uniform distribution over the path for this optimization.Source: §2.1 · p. 5
  • Robust mode connectivity uses adversarial training to find tunnels robust to different attack types, while connectivity claims in over-parameterized networks have stated exceptions.The section also cites applications to repairing backdoored or error-injected models, pretrained vision models, and language-model configurations.Source: §2.1 · p. 6

2.2 Non-linear Mode Connectivity

  • Nonlinear paths can connect minima outside the same basin when linear connectivity is unavailable, though some paths are difficult to optimize and approximations can encounter high-loss saddle points.Mechanistic similarity is associated with linear connectivity; relatively nonlinear paths may connect mechanistically dissimilar minima.Source: §2.2 · p. 7

2.3 Mode Connectivity in Subspace

  • Subspace methods seek low-loss connectors in lower-dimensional structures, including multidimensional manifolds and geodesics for narrow architectures where general tunnels may cross high-loss regions.The results can depend on subspace properties, and the section describes solutions in random low-dimensional hyperplanes when points are sufficiently dense in most cases.Source: §2.3 · pp. 7–8

2.4 Discussion

  • Mode connectivity can provide a better-performing model as a starting point for fusion, potentially saving time and compute and suiting data-limited settings.Connecting models can also add complexity and flexibility that increase overfitting risk; fine-tuning or parameter changes may increase training time and resource use.Source: §2.4 · pp. 8–9

3 Alignment

Alignment adjusts model parameters or matches units to reduce differences between models before fusion. The section presents Re-basin, activation matching, and weight matching, and discusses their benefits and computational limits.

3 Alignment

  • Alignment matches units across models to reduce interference between components and provide better initial conditions for fusion.The section divides alignment into activation matching and weight matching, depending on whether the target is data-driven.Source: §3 · p. 9

3.1 Re-basin

  • Re-basin uses permutation symmetry to transport solutions into a common basin, but efficiently searching all possible permutations remains challenging.Permuting hidden-layer units can preserve a network’s function, enabling equivalent solutions in different parts of weight space.Source: §3.1 · p. 10

3.2 Activation Matching

  • Activation matching minimizes costs between units’ activations using assignment methods, but linear formulations can ignore second-order weight proximity and edge information.The described costs include cross-correlation, mutual information, ℓ2 distance, KL divergence, and Wasserstein distance.Source: §3.2 · pp. 10–11

3.3 Weight Matching

  • Weight matching aligns models without data distributions, but its assignment problems can be difficult to solve and some approaches are computationally expensive.Methods include LAP-based matching, permutation search, iterative equivalent-model optimization, Gromov-Wasserstein barycenters, and graph matching.Source: §3.3 · pp. 11–12

3.4 Discussion

  • Alignment can improve information sharing and generalization, while its combinatorial optimization adds computational overhead, especially for large deep models.The survey also identifies transfer learning as a potential application, where alignment can reduce source–target differences and improve learning on new domains.Source: §3.4 · pp. 12–13

4 Weight Average

Weight averaging combines model parameters, but it works best when models share a training trajectory or basin; the section surveys direct, iterative, and subspace approaches alongside model soups and task arithmetic. It also highlights the trade-off between efficient fusion and the constraints of model similarity, computational resources, and weight selection.

4 Weight Average

  • Weight averaging is most reliable for models with similar weights that share part of a training trajectory or lie in the same basin.Averaging models with widely varying weights can perform poorly; convex combinations within a shared basin may improve generalization over basin endpoints.Source: §4 · p. 13, §4.1 · p. 13

4.1 Weight Average

  • Direct weight averaging can fail for dissimilar models, while Fisher merging and RegMean provide alternative parameter-aggregation formulations.RegMean uses the linear-layer input inner-product matrix in the forward pass, which the survey says improves operational efficiency compared with Fisher averaging.Source: §4.1 · pp. 13, 15
  • Iterative averaging smooths parameter updates, but typically requires shared training history and depends on the averaging schedule.Geometric Polyak–Ruppert averaging is presented for poor training conditions, whereas tail averaging is suited to good data conditions and averages checkpoints near the end of training.Source: §4.1 · p. 15

4.3 Model Soup

  • 90.94% on ImageNet-1K: model soup surpasses the reported CoAtNet-7 result of 90.88% and ViT-G result of 90.45%.Model soup averages models fine-tuned with different hyperparameters and reduces the inference time required for ensemble learning.Source: §4.3 · p. 18

4.4 Model Arithmetic

  • Task arithmetic combines fine-tuning task vectors with a pretrained model to support multi-task performance and direct model editing.The survey describes LoRA Hub as ensembling LoRA modules trained on different tasks, and task arithmetic as a way to preserve pretrained-model behavior without expensive joint fine-tuning.Source: §4.4 · p. 19

4.5 Average in Subspace

  • Subspace averaging constrains training trajectories to lower-dimensional spaces to reduce computational burden, but projection and basis costs remain challenges.The survey notes that a gradient-projection matrix can be too large for a single GPU and calls for further study of subspace averaging mechanisms.Source: §4.5 · pp. 19–20

4.6 Discussion

  • Weight averaging is more promising when models share optimized trajectories or a pretrained starting point, but choosing averaging weights can require subjective, costly trials.The discussion also identifies reducing iteration-to-iteration weight fluctuation and improving stability and convergence as potential uses.Source: §4.6 · pp. 20–21

5 Ensemble Learning

Ensemble learning combines model outputs, while model reuse draws on pretrained models for new tasks. The survey presents their capacity to use diverse or heterogeneous models alongside the inference costs of ensembles and limitations in reuse methods.

5.1 Ensemble Learning

  • Ensemble learning combines predictions from multiple models, and diverse predictions can improve performance and reduce variance.The survey describes snapshot ensembles and FGE as ways to collect predictions from models along learning-rate cycles; overly similar models may provide little benefit.Source: §5 · p. 21, §5.1 · p. 21

5.2 Model Reuse

  • Model reuse uses pretrained models or their features, labels, or modalities for a new task, reducing reliance on training data or retraining from scratch.The survey describes fixed model reuse, Bag of Experts, PM2R, and methods for selecting useful source models; it notes that some reuse methods apply only in limited scenarios.Source: §5.2 · pp. 22–23

5.3 Discussion

  • Ensembles can combine heterogeneous classifiers without the parameter and size requirements of some fusion methods, but require running multiple models at inference.The survey identifies this runtime burden as a poor fit for applications with limited computational resources and costs.Source: §5.3 · p. 23

6 Application

The survey reviews model-fusion applications in federated learning, fine-tuning, distillation, and foundation models/LLMs. These applications use different aggregation strategies and face practical constraints such as heterogeneity, cost, and information conflicts.

6.1 Federated Learning

  • Federated learning aggregates participant updates into shared models; FedAvg is described as influential for its efficient use of computing resources, while One-shot reports a 51.5% relative gain over baseline on AUC.The survey illustrates both centralized server aggregation and decentralized aggregation between clients, and notes that heterogeneous models can make simple weight aggregation harmful.Source: §6.1 · pp. 23–25, Figure 6 · p. 24

6.2 Fine-tuning

  • Fine-tuning fusion includes averaging related models and blending zero-shot with fine-tuned weights; WiSE-FT targets distribution-shift accuracy while retaining target-distribution accuracy.Lo-fi independently fine-tunes nodes before averaging them, while auxiliary-task fine-tuning is used to improve out-of-distribution generalization.Source: §6.2 · pp. 26–27, Figure 7 · p. 26
  • Fine-tuned-model averaging can reduce training time and improve accuracy and generalization, but the combination with fine-tuning can add overhead and raise checkpoint-preservation or catastrophic-forgetting concerns.The survey notes that these concerns are especially relevant to transfer learning.Source: §6.2 · p. 27

6.3 Distillation

  • Distillation aggregates either multiple teachers into a student or multiple students after teacher-guided learning; teacher conflicts can leave some information overlooked.For the student-aggregation route, co-distillation synchronizes teacher and student outputs on the same sample, while FD periodically uploads average logits to address that requirement.Source: §6.3 · p. 28

6.4 Model Fusion on Foundation Models/LLMs

  • Foundation-model fusion includes Bayesian model selection, aligned-channel weight aggregation, evidence fusion, candidate ranking, and combinations of LoRA modules for downstream adaptation.The survey reports that Summa Fusion improves little without filtering poor-quality candidates, while LLM-BLENDER ranks candidates using PairRanker.Source: §6.4 · p. 29
  • Fine-tuning large foundation models is associated with improved distribution-shift robustness, and task arithmetic's negative task vector reduces toxic GPT-2 generations from 4.8% to 0.8%.Branch-Train-Merge is described as reducing the synchronization required for parallel LLM training.Source: §6.4 · p. 29

7 Conclusion

The conclusion summarizes deep model fusion through four categories and reviews its applications and engineering prospects. It identifies further work on fusion strategies, network relationships, adaptation, and practical effects.

7 Conclusion

  • The survey categorizes deep model fusion as mode connectivity, alignment, weight average, and ensemble learning, covering both model-weight fusion and output fusion.It compares methods by algorithm design, performance, advantages, and disadvantages, and discusses applications in federated learning, distillation, and LLMs.Source: §7 · p. 29
  • Future work should explore new aggregation patterns, better initial conditions, diverse ensemble frameworks, network-component relationships, and adaptive methods for heterogeneous models and complex scenarios.The conclusion also calls for attention to practical effects to promote development and application.Source: §7 · p. 29
Loading 2309.15698v1…