PaperlayerView exact revision on arXiv

Source-linked AI summary

Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks

MohammadReza Davari, Eugene Belilovsky

arXiv:2312.06795v2cs.LG

TL;DR

Merging fine-tunings of the same foundation model is a way to reuse models across tasks, but joint training can be constrained by data privacy and computation, and available fine-tuned models remain underused. Model Breadcrumbs merges those models using masked weight differences, and the paper reports improved multi-task performance with stable, generalizable hyperparameters.

  • Problem

    Joint training across tasks can be constrained by data privacy and computational demands, while publicly available fine-tuned models sharing a foundation architecture remain largely untapped.

    Source: §1 · p. 2

  • Method

    Model Breadcrumbs builds multi-task models from pre-existing fine-tuned foundation models without further training, using a masking framework to filter weight changes.

    Source: §3 · pp. 4–5

  • Results

    Model Breadcrumbs enhances performance across multiple tasks and is reported to have stable, generalizable hyperparameters.

    Source: §1 · pp. 2–3, §5 · p. 15

  • Takeaways & Limitations

    The authors present Model Breadcrumbs as a practical approach for multi-task learning and community-driven refinement using publicly available fine-tuned models.

    Source: §5 · p. 15

    Performance depends on the quality of the initial fine-tuned models, so poor generalization or severe overfitting can propagate.

    Source: §5 · p. 15

Abstract

from arXiv · show

The rapid development of AI systems has been greatly influenced by the emergence of foundation models. A common approach for targeted problems involves fine-tuning these pre-trained foundation models for specific target tasks, resulting in a rapid spread of models fine-tuned across a diverse array of tasks. This work focuses on the problem of merging multiple fine-tunings of the same foundation model derived from a spectrum of auxiliary tasks. We introduce a new simple method, Model Breadcrumbs, which consists of a sparsely defined weight set that guides model adaptation within the weight space of a pre-trained model. These breadcrumbs are constructed by subtracting the weights from a pre-trained model before and after fine-tuning, followed by a sparsification process that eliminates weight outliers and negligible perturbations. Our experiments demonstrate the effectiveness of Model Breadcrumbs to simultaneously improve performance across multiple tasks. This contribution aligns with the evolving paradigm of updatable machine learning, reminiscent of the collaborative principles underlying open-source software development, fostering a community-driven effort to reliably update machine learning models. Our method is shown to be more efficient and unlike previous proposals does not require hyperparameter tuning for each new task added. Through extensive experimentation involving various models, tasks, and modalities we establish that integrating Model Breadcrumbs offers a simple, efficient, and highly effective approach for constructing multi-task models and facilitating updates to foundation models.

1 Introduction

Joint training across many tasks can be impractical when data are private or computation is limited, while many publicly available fine-tuned models remain unused. Model Breadcrumbs reuses those models to build multi-task models and reports scalable, generalizable hyperparameters.

1 Introduction

  • Joint training can be infeasible when privacy restricts access to training data, and simultaneous training across many tasks can be computationally demanding.The paper notes these constraints even when fine-tuned models themselves are publicly accessible.Source: §1 · p. 2
  • Many publicly available fine-tuned models share a foundation-model architecture but remain largely untapped beyond inference.The paper presents this pool as a potential resource for model reuse.Source: §1 · p. 2
  • Task Arithmetic merges scaled task-vector differences but faces growing noise and validation-based hyperparameter tuning as task counts increase.The paper describes per-task validation and rising computational impracticality at scale.Source: §1 · p. 2
  • Model Breadcrumbs builds multi-task models from existing fine-tuned models and reports competitive performance with hyperparameters that generalize as task counts increase.The paper also describes the approach as simple and scalable, and reports robustness to hyperparameter variations.Source: §1 · pp. 2–3

2 Related Work

Related work includes merging models with different initializations, averaging fine-tuned models from a shared initialization, and methods for task-vector merging. Model Breadcrumbs is distinguished by its focus on scalable multi-task merging from a shared foundation model.

Model Merging

  • Model merging from different initializations often requires aligning networks to a reference, whereas this work merges models sharing a foundation-model initialization and studies scaling to many models.The cited alignment work includes [1] [53].Source: §2 · p. 3
  • Linear interpolation of models fine-tuned from the same pre-trained model can outperform both original models.This result motivated later work on merging fine-tuned models from a shared foundation model.Source: §2 · p. 3
  • Prior approaches include weighted averaging of same-dataset fine-tunes, improving a pre-trained model for downstream use [6], and Fisher-weighted merging across datasets [36, 37].The paper says Fisher computation and hyperparameter selection become increasingly expensive as more models are merged.Source: §2 · p. 3
  • Task Arithmetic forms task vectors and adds their scaled sum to pre-trained weights, but requires validation data for each new task and accumulates noise as tasks are added [21].The paper characterizes the validation requirement as computationally burdensome at scale.Source: §2 · p. 4
  • TIES masks low-magnitude weights and aligns signs, while Model Breadcrumbs masks both large and small magnitudes layer-wise; federated-learning work instead aggregates clients on one downstream task [29] [40].The paper notes that overall masking of small weights typically targets early layers.Source: §2 · p. 4

3 Model Breadcrumbs Framework

Model Breadcrumbs constructs sparse task vectors from pre-existing fine-tuned foundation models, filtering extreme weight changes before combining them with the pretrained model. Its pipeline uses layer-wise masks to exclude both large outliers and negligible perturbations.

3 Model Breadcrumbs Framework

  • Model Breadcrumbs constructs multi-task models from pre-existing fine-tuned foundation models without further training, aiming to aggregate useful knowledge while filtering harmful perturbations.The framework describes the resulting sparse weight sets as trajectories through the foundation model's weight space.Source: §3 · pp. 4–5
  • Each task vector is formed from the weight difference between a fine-tuned model and its pretrained foundation model.The overview identifies these differences as fine-tuning trajectories.Source: §3 · p. 4, Figure 1 · p. 5
  • Masking removes both large outliers and negligible perturbations, which the paper identifies as extremes that can affect merged-model effectiveness.The masks are applied to absolute task-vector magnitudes within each layer.Source: §3 · p. 5
  • For each layer, the mask uses β and γ thresholds on weights sorted by absolute magnitude to exclude the lower and upper tails.The paper defines the layer mask using the sorted weight indices and the two thresholds.Source: §3 · p. 5
  • The masked task vectors are aggregated across tasks with strength parameter α and combined with the reference pretrained weights to form the multi-task model.The paper describes this as traversing the filtered task trajectories.Source: §3 · p. 5

4 Experiments

Experiments compare Model Breadcrumbs with other merging methods across vision and NLP tasks, model scales, and task counts. They assess performance, hyperparameter generalization, target-task improvements, and design choices.

4.2 Merging Model Breadcrumbs

  • Across 256 task subsets, Model Breadcrumbs outperform Task Arithmetic; on all eight tasks, the margin is 5.7%, and they also lead Fisher Merging in the eight-task ViT-B-32 comparison.The subset evaluations consider both all eight tasks and only tasks already observed; the gap over Task Arithmetic grows as more tasks are merged.Source: Table 1 · p. 7, §4.2 · p. 8, Figure 2 · p. 8

4 Experiments; 4.1 Data, Metrics, and Models

  • The experiments use normalized accuracy relative to fine-tuned models and span eight vision datasets with three CLIP scales, plus four GLUE tasks using T5-base.The study also examines task-count scalability, scale, target-task improvement, and ablations of the merging design.Source: §4 · p. 6, §4.1 · p. 7

4.3 Validation-Free Setting

  • With hyperparameters fixed after 1–3 tasks for ViT-B-32 or one task for ViT-L-14, Breadcrumbs generalize through eight tasks; after tuning on 10 ImageNet tasks, they outperform Task Arithmetic across 200 5-class tasks.The authors say this stability lets practitioners select hyperparameters from relatively few tasks, reducing extensive tuning.Source: Figure 3 · p. 9, §4.3 · pp. 9–10, Figure 4 · p. 10

4.4 Effect of Scale

  • On ViT-L-14 with eight tasks, Breadcrumbs reach 91.48% of the performance of eight individually fine-tuned models; larger models mitigate merging degradation, and pairwise merges often improve both tasks.For some task pairs, Breadcrumbs produce models that match or surpass individual fine-tuned accuracy on both tasks more often than Task Arithmetic.Source: §4.4 · pp. 10–12, Figure 6 · p. 11

4.5 Target Task Improvement via Model Merging

  • The reported results show that incorporating Breadcrumbs from newly added tasks into fine-tuned models improves performance, surpassing all other considered approaches, without additional training or requiring data from the exact same dataset.Table 2 reports average performance over 20 runs, followed by the standard error.Source: §4.5 · p. 12, Table 2 · p. 12

4.6 Ablations

  • Combining top- and bottom-weight masking is the most effective design; its hyperparameter search yields more high-performing models and less performance variation than the alternatives.The ablation compares full Breadcrumbs with bottom-only masking, top-only masking, and unmasked Task Arithmetic.Source: §4.6 · pp. 12–14, Figure 7 · p. 13
  • Breadcrumbs move task cosine similarities closer to zero, while semantically similar tasks can retain higher similarity under Task Arithmetic, potentially introducing interference during merging.The authors suggest reduced interference could explain the improved performance of merged Breadcrumbs models.Source: Figure 8 · p. 13, §4.6 · p. 14
  • Across eight vision tasks with ViT-B-32, increasing β requires lower α, while the optimal γ tends to stay near 1%; many parameter combinations still yield high-performing models.β and γ denote how much weight is masked in this analysis.Source: §4.6 · p. 14, Figure 9 · p. 14

5 Conclusions

Model Breadcrumbs constructs multi-task models from pre-existing fine-tuned foundation models and is reported to enhance performance across tasks with stable, generalizable hyperparameters. Its performance depends on the quality of the initial fine-tuned models, and the authors connect the method to community-driven model refinement.

5 Conclusions

  • Model Breadcrumbs enhances performance across multiple tasks with stable, generalizable hyperparameters, and larger models further benefit from the approach.The conclusion also notes that NLP experiments highlight versatility across modalities.Source: §5 · p. 15
  • Performance depends on the quality of the initial fine-tuned models, as poor generalization or severe overfitting can propagate.The authors identify model-capacity expansion as important for sustained high performance as task numbers increase.Source: §5 · p. 15
  • The method leverages publicly available fine-tuned models to support community-driven model refinement and efficient, scalable multi-task learning.The authors align this approach with updatable machine learning.Source: §5 · p. 15

A Vision: Data and Training

The vision experiments fine-tune CLIP models with a specified optimization setup, while freezing the text encoder classification layer to avoid adding learnable parameters.

A Vision: Data and Training

  • CLIP models are fine-tuned for 2,000 iterations with batch size 128, learning rate 1e-5, cosine annealing, 200 warm-up steps, and AdamW weight decay 0.1.Datasets are specified in Section 4.1; the passage also refers to Table 3 for details.Source: A Vision: Data and Training · p. 20
  • The CLIP text encoder classification layer is frozen, avoiding additional learnable parameters, as in prior work.Source: A Vision: Data and Training · p. 20

B NLP: Data and Training

The NLP experiments fine-tune T5-base on GLUE tasks and use publicly available fine-tuned T5 models for additional tasks.

B NLP: Data and Training

  • T5-base is fine-tuned on MRPC, RTE, CoLA, and SST-2 for five epochs with batch size 32, learning rate 1e-5, and AdamW.A validation set equal in size to the test set is created from training data to select the best model.Source: B NLP: Data and Training · p. 20
  • Publicly available T5-base models are sourced from Hugging Face for IMDB, RACE, QASC, MultiNews, SQuAD, and CommonGen.Source: B NLP: Data and Training · p. 20

C TIES Merging

Across the evaluated task counts, Model Breadcrumbs consistently outperforms TIES, with the gap widening as tasks increase. The methods differ in which weights they mask and how masking is applied.

C TIES Merging

  • TIES masks low-magnitude weights, aligns the signs of the remaining weights, then merges task vectors with the pre-trained model using a scaled sum.Source: C TIES Merging · pp. 20–21
  • Model Breadcrumbs consistently outperforms TIES, with the performance gap widening as the number of tasks increases.Figure 10 averages normalized accuracy across all subsets of eight tasks (256 combinations) and evaluates all eight tasks at each point.Source: Figure 10 · p. 21, C TIES Merging · p. 21
  • Model Breadcrumbs masks both very large and small task-vector weights layer-wise, whereas TIES masks only small-magnitude weights using overall masking.Overall masking ranks weights across the model and typically targets early layers in task vectors.Source: C TIES Merging · p. 21
Loading 2312.06795v2…