PaperlayerView exact revision on arXiv

Source-linked AI summary

Composable Sparse Fine-Tuning for Cross-Lingual Transfer

Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, Ivan Vulić

arXiv:2110.07560v2cs.CL

TL;DR

Full-model fine-tuning is inefficient and can cause catastrophic forgetting and interference, while adapters and sparse fine-tuning each provide only part of the desired modularity–expressivity combination. The paper introduces LT-SFT, whose composable sparse adaptations consistently outperform baselines in zero-shot cross-lingual transfer, while sparsity remains important for avoiding interference and overfitting.

  • Problem

    Full-model fine-tuning is time- and resource-inefficient and may cause catastrophic forgetting and interference, motivating methods that combine adapters’ modularity with sparse fine-tuning’s expressivity.

  • Method

    LT-SFT selects the parameters that change most during task or language adaptation, rewinds to pretrained initialization, and re-tunes only those parameters so sparse adaptations can be composed.

  • Results

    LT-SFT consistently outperforms baselines across four multilingual tasks, including gains over MAD-X of 2.5 accuracy points in part-of-speech tagging and 1.8 F1 points in named entity recognition.

  • Takeaways & Limitations

    Sparsity is crucial for composable adaptation because it helps prevent interference between fine-tunings, although excessive capacity and overfitting may also contribute to degradation.

  • Takeaways & Limitations

    The explanation for performance degradation remains inconclusive because parameter overlap and overfitting from excessive capacity are alternative hypotheses.

Abstract

from arXiv · show

Fine-tuning the entire set of parameters of a large pretrained model has become the mainstream approach for transfer learning. To increase its efficiency and prevent catastrophic forgetting and interference, techniques like adapters and sparse fine-tuning have been developed. Adapters are modular, as they can be combined to adapt a model towards different facets of knowledge (e.g., dedicated language and/or task adapters). Sparse fine-tuning is expressive, as it controls the behavior of all model components. In this work, we introduce a new fine-tuning method with both these desirable properties. In particular, we learn sparse, real-valued masks based on a simple variant of the Lottery Ticket Hypothesis. Task-specific masks are obtained from annotated data in a source language, and language-specific masks from masked language modeling in a target language. Both these masks can then be composed with the pretrained model. Unlike adapter-based fine-tuning, this method neither increases the number of parameters at inference time nor alters the original model architecture. Most importantly, it outperforms adapters in zero-shot cross-lingual transfer by a large margin in a series of multilingual benchmarks, including Universal Dependencies, MasakhaNER, and AmericasNLI. Based on an in-depth analysis, we additionally find that sparsity is crucial to prevent both 1) interference between the fine-tunings to be composed and 2) overfitting. We release the code and models at https://github.com/cambridgeltl/composable-sft.

1 Introduction

The paper seeks a parameter-efficient method that combines adapters’ modularity with sparse fine-tuning’s expressivity. It introduces LT-SFT, which composes sparse task and language adaptations and consistently outperforms adapter baselines in multilingual zero-shot transfer.

  • Motivation: Adapters provide modular language and task specialization, whereas sparse fine-tuning directly controls pretrained model components.This motivates combining modularity with expressivity in one parameter-efficient method.
  • Method: LT-SFT selects the most changed parameters, rewinds to pretrained initialization, and retunes only that subset to create sparse difference vectors.These vectors can be summed with the pretrained model and with one another.
  • Evaluation: LT-SFT is evaluated on Universal Dependencies, MasakhaNER, and AmericasNLI in zero-shot transfer across 35 diverse languages.The languages include both those seen and unseen during pretrained-model masked language modeling.
  • Results: LT-SFT consistently outperforms baselines across multilingual transfer tasks, while retaining modularity and expressivity.The reported comparisons include gains over MAD-X across all four evaluated tasks.
  • Analysis: High sparsity reduces overlap and interference between language and task fine-tunings while limiting overfitting, supporting compositional generalization.The paper presents sparsity as fundamental to modularity and zero-shot combinations of tasks and languages.

2 Background

The background reviews adapters, sparse fine-tuning, and the Lottery Ticket Hypothesis as foundations for composable adaptation. It contrasts adapter composition with sparse difference-vector methods and motivates applying lottery-ticket selection to pretrained-model adaptation.

  • Adapters and Composition: Adapters insert task-, language-, domain-, or modality-specific modules into Transformer layers while keeping pretrained parameters fixed.The reviewed Pfeiffer-style adapter uses down- and up-projections around a nonlinear activation.
  • Adapters and Composition: MAD-X composes separately trained language and task adapters by stacking the target-language adapter with the task adapter at inference.Language adapters are trained with masked language modeling, while task adapters use task data.
  • Sparse Fine-Tuning: Sparse fine-tuning represents adaptation as a sparse difference vector φ added to pretrained parameters θ.Prior methods include DiffPruning, BitFit, and ChildTuning; some remain competitive with full fine-tuning using fewer than 0.5% non-zero values.
  • Lottery Ticket Hypothesis: The Lottery Ticket Hypothesis proposes that a neural model contains a trainable subnetwork that can match or exceed the original model after pruning, restoration, and retuning.The paper applies this idea to pretrained-model adaptation rather than only model compression.
  • Related Work: Multi-source task training can improve downstream zero-shot transfer when batches draw examples from randomly selected source languages.The cited setup activates the corresponding language adapter during each training step.

3 Methodology

LT-SFT identifies a sparse subset of parameters through a two-phase Lottery Ticket procedure, then learns composable language- and task-specific sparse fine-tunings. These components are combined for zero-shot transfer while preserving the pretrained model architecture.

  • Lottery Ticket Sparse Fine-Tuning: LT-SFT first fully fine-tunes a pretrained model, selects the K parameters with the greatest absolute changes, resets the model, and retrains only those parameters.The resulting sparse difference vector is obtained relative to the pretrained parameters.
  • Composition: SFT composition adds sparse difference vectors to the pretrained parameter vector, allowing language and task adaptations to be combined without adding adapter layers.The method conceptualizes an SFT as a function that transforms a parameterized model by adding a sparse difference vector.
  • Language and Task SFTs: Language SFTs are learned with masked language modeling on each language, while task SFTs are learned from annotated source-language data.Task training first applies the source-language SFT, which is removed after training.
  • Transfer Effects: Source-language adaptation improves cross-lingual transfer by 2-3 points in preliminary experiments for both MAD-X and LT-SFT.The authors report that adapting high-resource target languages at inference time does not yield similarly large benefits.
  • Zero-Shot Transfer: Zero-shot transfer composes a target-language SFT with a task SFT and adds the task classifier head.The composed model is evaluated on the target language without target-language task annotations.

4 Experimental Setup

The experiments evaluate LT-SFT against adapter baselines across multilingual zero-shot transfer tasks, emphasizing diverse, low-resource, and unseen languages. Training uses matched parameter budgets, task-specific datasets, held-out validation, and broad hyperparameter sweeps.

  • Evaluation Scope: The evaluation covers POS tagging, dependency parsing, NER, and NLI across 35 typologically and geographically diverse languages.The setup emphasizes low-resource languages and languages unseen during multilingual pretraining.
  • Baselines and Variants: MAD-X 2.0 is the main adapter baseline, and both methods are also tested in task-only configurations without target-language adaptation.The authors train adapters for all evaluation languages because configurations from prior work are unavailable for many of them.
  • Data: Language adaptation uses Wikipedia or task-specific multilingual corpora, while task adaptation draws on datasets for POS, dependency parsing, NER, and NLI.The experiments also include multi-source task training for dependency parsing, NLI, and extractive question answering.
  • Training Procedure: Training uses held-out validation and checkpoint selection by loss or task metric, with separate optimization settings for language adaptation and the evaluated tasks.Language and task procedures differ in epochs, batch sizes, learning rates, and validation intervals.
  • Parameter Budgets: The adapter baseline has approximately 7.6M trainable parameters, and language LT-SFTs use the same K for comparability.This yields language SFT sparsities of 4.3% for mBERT and 2.8% for XLM-R.
  • Language SFT Details: Language SFT training fixes output embeddings and layer-normalization parameters to avoid selecting mostly embedding parameters, which damages downstream performance.Input and output embeddings are decoupled during language sparse fine-tuning.

5 Results and Discussion

LT-SFT consistently outperforms the evaluated baselines in zero-shot cross-lingual transfer across four tasks, while language adaptation improves transfer to unseen languages. Multi-source training also helps, and performance declines when fine-tunings become too dense.

  • Overall results: LT-SFT consistently outperforms all baselines across four zero-shot cross-lingual transfer tasks, including MAD-X and RAND-SFT.The comparison covers part-of-speech tagging, dependency parsing, named entity recognition, and natural language inference.
  • Overall results: 2.5 accuracy in part-of-speech tagging, 2.5 UAS and 3.7 LAS in dependency parsing, 1.8 F1 score in named entity recognition, and 1.9 accuracy in natural language inference are LT-SFT’s gains over MAD-X.These results are averaged over languages using the best reduction factor.
  • Language adaptation: Language SFTs or adapters produce large gains across all four tasks compared with task adaptation only, especially for specialization to unseen languages.The results identify language adaptation as important for zero-shot transfer to languages unseen during masked language modeling.
  • Overall results: 51.4% average accuracy on AmericasNLI exceeds the 48.7% of the translate-train baseline in zero-shot transfer.This result shows that LT-SFT also surpasses a translation-based baseline on natural language inference.
  • Parameter scaling: LT-SFT and RAND-SFT generally improve or remain steady as trainable task parameters increase, whereas MAD-X can degrade at lower reduction factors.Unlike MAD-X, the SFT reduction-factor choice does not affect inference time.
  • Multi-source training: Multi-source LT-SFT substantially improves dependency parsing and modestly improves natural language inference in zero-shot cross-lingual transfer.The difference may relate to the smaller number of non-English NLI training examples and lower genealogical relatedness of AmericasNLI languages.
  • Multi-source training: Multi-source LT-SFT training for XLM-R Base outperforms single-source full fine-tuning of both XLM-R Large and XLM-R Base on cross-lingual question answering.This occurs despite each non-English source language having more than an order of magnitude less training data than English SQuAD.
  • Sparsity and interference: Performance decreases markedly when SFT density exceeds approximately 30%, consistent with interference or excessive-capacity explanations.The analysis leaves conclusive evidence distinguishing parameter overlap from overfitting for future work.

6 Related Work

Prior work extended the Lottery Ticket Hypothesis to stabilize winning-ticket discovery, prune NLP and Transformer models, and compose task-specific masks for continuous learning.

  • Lottery Ticket Hypothesis improvements include rewinding parameters or learning rates and selecting weights with criteria beyond magnitude of change.Frankle et al. rewind to values after a few iterations, Renda et al. also rewind the learning rate, and Zhou et al. identify alternative selection criteria.
  • Winning tickets have been applied to NLP pruning, including neural machine translation and Transformer-based models.
  • Wortsman et al. previously composed task-specific winning-ticket masks by linearly combining them at inference time for continuous learning.

7 Conclusion and Future Work

The paper presents LT-SFT as a modular and expressive fine-tuning method that composes sparse language and task adaptations for zero-shot cross-lingual transfer, with substantial gains across multilingual tasks.

  • LT-SFT combines modularity with expressivity by inferring sparse language and task difference vectors relative to the pretrained model.Language vectors are learned from unlabeled text, while task vectors use supervised learning.
  • The method and its algorithm use sparse difference vectors and return the pretrained parameters plus task and target-language adaptations.The algorithm first learns a source-language adaptation, then a task adaptation, and separately learns a target-language adaptation.
  • Language and task adaptations are composed with the pretrained model to enable zero-shot cross-lingual transfer.
  • LT-SFT yields substantial gains across several multilingual tasks for languages seen and unseen during pretraining, including truly low-resource languages.
  • Future work proposes alternative pruning-based selection algorithms and applications beyond cross-lingual transfer, including multimodal learning, debiasing, and domain adaptation.The code and models are released online.

B Languages

The paper documents languages and data used for SFT and adapter training and identifies source languages used in multi-source task-training experiments.

  • Table 5 provides details of languages and data used to train and evaluate SFTs and adapters.The caption references multilingual data repositories for NLI corpora and the corpora of Bustamante et al. (2020).
  • English is the source language in all single-source task-training experiments.
  • The language annotations distinguish source languages for multi-source dependency parsing, natural language inference, and question answering training.

C Results by Language

The results tables report zero-shot cross-lingual transfer performance by language, covering multilingual tasks and distinguishing seen, high-resource, and resource-poor evaluation languages.

  • Table 6 reports zero-shot cross-lingual transfer results across all tasks for each language and identifies the best-performing method per language.LT-SFT with multi-source training is excluded from boldface because its larger, more diverse dataset gives it an unfair advantage.
  • Table 7 reports zero-shot transfer results for seen languages in POS, dependency parsing, and named entity recognition evaluations.Arabic, Japanese, and Chinese are treated as high-resource languages, while Swahili and Yorùbá are described as resource-poor.

D MAD-X Results with AdapterHub Adapters

LT-SFT maintains an edge over MAD-X across all evaluated tasks, despite AdapterHub adapters receiving substantially more training steps and a larger batch size.

  • LT-SFT maintains an edge in performance across all tasks compared with MAD-X using AdapterHub language adapters.The comparison uses AdapterHub adapters during task training and evaluation.

E Parameter Overlap between Languages

Language-specific sparse fine-tuning masks have small parameter overlap for most language pairs, with Mandarin Chinese and Cantonese as a notable exception linked to genealogical relatedness.

  • Most language pairs show small overlap between their language-specific parameter subsets.The overlap is measured pairwise as a percentage of shared selected parameters.
  • Mandarin Chinese and Cantonese are the sole reported exception, exhibiting high overlap that reflects their genealogical relationship.
Loading 2110.07560v2…