Source-linked AI summary
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Elad Ben-Zaken, Shauli Ravfogel, Yoav Goldberg
TL;DR
Fine-tuning large pretrained transformers is costly, motivating methods that change few parameters while retaining task performance. The paper introduces BitFit, which trains only bias terms, and finds that it matches or sometimes exceeds full fine-tuning on small-to-medium datasets while remaining competitive with other sparse methods. These results also support viewing fine-tuning as exposing pretrained knowledge rather than primarily learning new task-specific linguistic knowledge.
Problem
Large pretrained transformer models are expensive to train and deploy, motivating fine-tuning methods that change fewer parameters while preserving performance.
Method
BitFit freezes most transformer parameters and trains only the bias terms and task-specific classification layer.
Results
For small to medium training data, BitFit reaches the same task accuracy as full fine-tuning and sometimes improves results; it also outperforms Diff-Pruning on 4 of 9 validation tasks with 6x fewer trainable parameters.
Takeaways & Limitations
BitFit eases deployment because most model parameters are shared across tasks and enables hardware implementations with most computations fixed.
Takeaways & Limitations
QNLI results are not directly comparable because the GLUE benchmark updated its test set.
Abstract
from arXiv · showhide
We introduce BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. For larger data, the method is competitive with other sparse fine-tuning methods. Besides their practical utility, these findings are relevant for the question of understanding the commonly-used process of finetuning: they support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.
1 Introduction
BitFit targets the cost and conceptual burden of fine-tuning large pretrained transformers by changing only a small, consistent subset of parameters. On small-to-medium datasets, this sparse approach matches full fine-tuning and can sometimes improve results, while an even smaller bias subset trades a slight performance degradation for extreme parameter efficiency.
- BitFit changes very few parameters per task, using the same parameter set across tasks and localizing those changes throughout the model.These properties support task-invariant, isolated, and localized fine-tuning.
- For small to medium training data, changing only the selected parameters reaches the same task accuracy as full fine-tuning and sometimes improves results.
- BitFit freezes most of the network and fine-tunes only bias terms, making sparse adaptation surprisingly effective.
- Fine-tuning only the query and middle-of-MLP bias components uses half of the bias parameters and 0.04% of all model parameters, with a small performance degradation.
- The method can ease deployment of multitask models in memory-constrained environments and support hardware implementations with most parameters fixed.The paper also identifies research questions about bias terms and fine-tuning dynamics.
2 Background: fine-tuning and parameter-efficient fine-tuning
Fine-tuning pretrained encoders is effective but creates a separate large model for each task, motivating methods that alter fewer parameters while preserving performance. BitFit and related approaches address this deployment and data-access burden through sparse, task-efficient adaptation.
- Standard transfer learning adds a task-specific classifier to a pretrained encoder and trains the entire encoder-classifier network end-to-end.
- Per-task fine-tuning produces a large unique model for each task, making changes harder to analyze and deployment harder as tasks accumulate.
- An ideal fine-tuning method matches full fine-tuning, changes few parameters, supports tasks arriving sequentially, and reuses the same changed parameters across tasks.
- The fine-tuning process raises whether it learns new capabilities or exposes capabilities acquired during language-model pretraining.
- Adapters and Diff-Pruning achieve sparse adaptation through task-specific modules or sparse parameter difference vectors, while BitFit performs favorably on many tasks and satisfies task-invariant parameter selection.
3 Bias-terms Fine-tuning (BitFit)
BitFit freezes most transformer-encoder parameters and fine-tunes only bias terms alongside the task-specific classifier. This parameter-efficient approach can match or sometimes exceed full-model fine-tuning, while a smaller bias subset can still rival it.
- Method: BitFit freezes most transformer-encoder parameters and trains only bias terms plus the task-specific classification layer.It is designed to support task streams without simultaneous access to all datasets.
- Method: The method stores only bias-term vectors and the task-specific final linear classifier for each new task.The stored bias vectors comprise less than 0.1% of the model’s total parameters.
- Method: Bias parameters make up 0.09% of BERTBASE and 0.08% of BERTLARGE parameters.The bias terms are additive components of the network.
- Results: Freezing all W(·) and g(·) parameters while fine-tuning only additive bias terms achieves performance comparable to, and sometimes better than, full-network fine-tuning.This is the central transfer-learning result of BitFit.
- Results: Fine-tuning only query and second-MLP-layer bias parameters still achieves accuracies that rival full-model fine-tuning.This subset uses fewer trainable parameters while retaining competitive accuracy.
4 Experiments and Results
Experiments show that BitFit performs strongly across evaluated tasks while updating only bias parameters, with especially favorable results in smaller-data settings. The experiments also identify which bias subsets matter and show that randomly selected parameters perform worse.
- BitFit outperforms Diff-Pruning on 4 out of 9 validation tasks while using 6x fewer trainable parameters, and records two clear test-set wins against Diff-Pruning and four against Adapters while using 45x fewer parameters.
- BitFit trends remain consistent across BERTLARGE, BERTBASE, and RoBERTaBASE.
- Randomly selected parameters perform substantially worse across tasks than the same-sized bias-parameter subset.
- Fine-tuning only the query and intermediate-MLP bias terms produces results only marginally below tuning all bias parameters, while either type alone performs substantially worse.
- BitFit models have a substantially smaller generalization gap than full-fine-tuning models, despite full fine-tuning often reaching nearly 100% training accuracy.
- On PTB POS-tagging, BitFit matches Full-FT for BERTBASE and BERTLARGE and reaches 97.1 versus 97.2 for RoBERTaBASE.
- In smaller-data SQuAD settings, BitFit dominates Full-FT, whereas Full-FT performs better as more training data becomes available.
5 Related Work
Related work frames sparse fine-tuning as both a model-compression problem and a way to investigate what pre-training and fine-tuning contribute. Prior methods modify adapters, sparse parameter differences, or selected bias-related components, while BitFit provides direct evidence for bias-only adaptation in transformers.
- Identifying the minimal fine-tuned parameter set connects practical model compression with questions about pre-training, fine-tuning, and transferable linguistic knowledge.
- Adapters add trainable task-specific modules, whereas Diff-Pruning adds sparse task-specific difference vectors while retaining shared original parameters.
- Prior work rarely discusses bias terms, and one masking-based fine-tuning method explicitly reported no positive performance effect from handling them.
- Related studies analyze bias attribution, personalize output vocabularies by tuning softmax biases, train batch-normalization layers, and adapt computer-vision models with bias-only fine-tuning.
- The Transformer paper’s equations omit bias terms, although BERT models include them.
6 Conclusions
BitFit offers localized fine-tuning by modifying a small bias subset while retaining most pretrained transformer parameters. The method supports shared deployment and hardware efficiency, and its effectiveness raises questions about fine-tuning dynamics and transfer.
- BitFit fine-tunes biases for end tasks while maintaining good performance across the evaluated GLUE tasks.
- Because most parameters remain shared across tasks, BitFit eases deployment of multi-task fine-tuned models.
- Keeping most pretrained weights fixed enables hardware implementations that hard-wire most computation and leave only a few changeable inference components.
- Bias-only fine-tuning raises questions about pretrained-transformer fine-tuning dynamics and the relationship between bias terms and transfer from language modeling to new tasks.
A.1 Layer naming
This appendix section maps the bias-parameter names used in HuggingFace’s BertLayer implementation to the notation used in the BitFit paper.
- The appendix provides this correspondence for convenience when relating the paper to the HuggingFace implementation.
- Table 4 maps HuggingFace BertLayer bias-parameter names to BitFit notation.
- The mapping connects implementation-specific parameter names with the paper’s bias notation.
A.2 Training Details
The experiments attach linear classifiers to BERT’s [CLS] representation and train with AdamW. Bias-only fine-tuning uses larger learning rates and generally converges within 8 epochs, while some tasks require up to 20.
- BERT classification attaches a linear layer to the [CLS] contextual embedding and uses standard GLUE input procedures.
- The experiments optimize with AdamW using batch sizes of 16.
- Bias-only fine-tuning uses learning rates from 1e-4 to 1e-3, while full fine-tuning uses 1e-5 to 5e-5.
- Bias-only fine-tuning converges in 8 or fewer epochs for most tasks and up to 20 epochs for others.
- Each evaluation reports the average and standard deviation across 5 models trained with 5 different random seeds.
A.3 GLUE Benchmark
The GLUE evaluation covers multiple classification and inference tasks using standard train:dev:test partitions and task-specific metrics. The appendix also lists the evaluated metrics and learning-rate configurations, alongside figures for bias changes and SQuAD validation performance.
- The evaluated GLUE subset includes CoLA, SST-2, MRPC, QQP, STS-B, MNLI, and QNLI.
- All experiments use the common GLUE train:dev:test partition.
- Table 5 lists the metrics used to evaluate the GLUE Benchmark.
- Table 6 lists learning-rate configurations for the best-performing models.
- Figures 3–5 show changes in bias components for CoLA, MRPC, and STS-B, while Figure 6 compares BitFit and Full-FT on SQuAD validation F1.