PaperlayerView exact revision on arXiv

Source-linked AI summary

LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning

Yi-Lin Sung, Jaemin Cho, Mohit Bansal

arXiv:2206.06522v2cs.CLcs.AIcs.CV

TL;DR

Large pre-trained models are costly to adapt because existing PETL methods still require backpropagation through the backbone, limiting memory savings. LST instead trains a separate ladder side network from intermediate backbone activations, and it delivers greater memory efficiency with competitive or better accuracy across NLP and vision-language tasks.

  • Problem

    Existing PETL methods reduce updated parameters but save training memory by only up to 30% because gradients still require backpropagation through the large backbone.

  • Method

    LST trains a small separate side network that receives intermediate backbone activations through ladder shortcuts while the backbone remains frozen.

  • Results

    LST saves 69% of full fine-tuning memory cost versus 26% for Adapter and LoRA at similar parameter usage, while achieving strong accuracy across NLP and vision-language tasks.

  • Takeaways & Limitations

    LST enables adapting larger pre-trained backbones under limited memory and provides a more efficient accuracy-memory trade-off than recent PETL baselines.

Abstract

from arXiv · show

Fine-tuning large pre-trained models on downstream tasks has been adopted in a variety of domains recently. However, it is costly to update the entire parameter set of large pre-trained models. Although recently proposed parameter-efficient transfer learning (PETL) techniques allow updating a small subset of parameters (e.g. only using 2% of parameters) inside a pre-trained backbone network for a new task, they only reduce the training memory requirement by up to 30%. This is because the gradient computation for the trainable parameters still requires backpropagation through the large pre-trained backbone model. To address this, we propose Ladder Side-Tuning (LST), a new PETL technique that can reduce training memory requirements by more substantial amounts. Unlike existing parameter-efficient methods that insert additional parameters inside backbone networks, we train a ladder side network, a small and separate network that takes intermediate activations as input via shortcut connections (called ladders) from backbone networks and makes predictions. LST has significantly lower memory requirements than previous methods, because it does not require backpropagation through the backbone network, but instead only through the side network and ladder connections. We evaluate our method with various models (T5 and CLIP-T5) on both NLP (GLUE) and vision-and-language (VQA, GQA, NLVR2 , MSCOCO) tasks. LST saves 69% of the memory costs to fine-tune the whole network, while other methods only save 26% of that in similar parameter usages (hence, 2.7x more memory savings). Moreover, LST achieves higher accuracy than Adapter and LoRA in a low-memory regime. To further show the advantage of this better memory efficiency, we also apply LST to larger T5 models, attaining better GLUE performance than full fine-tuning and other PETL methods. The accuracy-efficiency trade-off also holds on VL tasks.

1 Introduction

Parameter-efficient transfer learning reduces the number of updated parameters but leaves substantial training-memory costs because gradients still require backpropagation through the backbone. Ladder Side-Tuning addresses this by training a separate ladder side network, achieving stronger memory savings and competitive accuracy across NLP and vision-language tasks.

  • Motivation: PETL methods update only a small subset of parameters, but their trainable parameters remain inside the backbone network.This preserves the need for backward computation through the large pre-trained model.
  • Method: LST trains a lightweight side network that receives intermediate backbone activations through shortcut connections and makes predictions.Its trainable parameters are separated from the pre-trained backbone.
  • Method: LST eliminates backpropagation through the backbone, substantially reducing training memory compared with adapters and prompt tuning.The backbone remains frozen while gradients are computed through the side network and ladder connections.
  • Results: LST achieves higher accuracy than other PETL methods in low-memory settings and remains competitive on diverse NLP and vision-language tasks.The evaluation covers GLUE, VQA, GQA, NLVR2, and MSCOCO, including larger T5 models.

2 Related Work

Prior PETL methods reduce parameter updates through adapters, prompts, or related mechanisms, while memory-efficient training and compression pursue different ways to reduce training or inference cost. LST combines parameter and memory efficiency and is presented as more flexible than Y-tuning for architectures and tasks where enumerating all answers is impractical.

  • Parameter-efficient transfer learning: Adapters insert small modules into transformer layers, while prompt methods add trainable parameters to the input and keep the backbone unchanged.LoRA instead injects trainable low-rank matrices into the frozen model.
  • Memory-efficient training: Parameter efficiency does not necessarily provide memory efficiency because many PETL methods still compute backbone intermediate-output gradients.LST is introduced to provide both benefits simultaneously.
  • Alternative methods: LST is more flexible than Y-tuning because listing all possible answers is often impractical for regression and open-ended generation tasks.The paper reports that LST outperforms Y-tuning with fewer parameter updates.
  • Memory-efficient training: Reversible networks and gradient checkpointing reduce activation storage, whereas LST keeps the backbone frozen and trains a separate side network.These memory-saving approaches can be combined with LST.
  • Network compression: Network compression reduces model size through pruning or distillation, and LST uses these techniques to initialize its side network.The paper primarily uses pruning because it is efficient under limited-resource constraints.

3 Ladder Side-Tuning (LST)

LST reduces training memory by separating trainable parameters from the frozen backbone into a lightweight ladder side network. Its design combines shortcut connections, structural initialization, and layer dropping to improve efficiency while retaining performance.

  • 3.1 Dependency on Backpropagation through Large Backbone Model: Backpropagation memory is dominated by activations for updated parameters and cached activation derivatives, limiting conventional PETL savings.Existing methods reduce the activation term but generally retain the derivative term because their trainable parameters remain inside the backbone.
  • 3.2 Ladder Side Network for Transformers: LST trains a separate ladder side network from intermediate backbone activations, avoiding backpropagation through the large backbone.The side network receives activations through shortcut connections and makes predictions independently of backbone parameter updates.
  • 3.2 Ladder Side Network for Transformers: A reduction factor r shrinks side-network weights and hidden states; for example, r = 16 changes a 768-dimensional backbone state to 48 dimensions.The side network reuses the backbone’s frozen word embeddings and language-model head.
  • 3.2 Ladder Side Network for Transformers: LST’s side-network gradients have a memory footprint reduced by factor r, yielding better efficiency than other PETL methods when r exceeds 2.The experiments find r = 8 works well in most cases.
  • 3.2 Ladder Side Network for Transformers: Gated ladder connections downsample backbone activations into low-dimensional side-network blocks and upsample side-network outputs to the original head dimension.The design addresses the observed weakness of late fusion for transformer architectures in NLP tasks.
  • 3.3 Structural Weight Initialization and 3.4 Layer Dropping: Structural pruning initializes the side network from selected backbone weights, while layer dropping linearly reduces memory and parameter requirements without sacrificing performance.Fisher information generally performs well as the importance metric for structural initialization.

4 Experiment Setup

The experiments evaluate LST against full fine-tuning and PETL baselines across GLUE and vision-and-language tasks using T5-based backbones. Metrics, splits, memory usage, and training settings are specified for reproducible comparisons.

  • Datasets: GLUE includes seven classification tasks and one regression task spanning acceptability, sentiment, similarity, paraphrase, and inference.
  • Baselines: The baselines include full fine-tuning, Adapters, and other parameter-efficient approaches, with full fine-tuning serving as the performance upper bound.
  • Training and Evaluation Setup: T5-base is the primary NLP backbone, while T5-large and T5-3B are used to assess LST scaling and memory efficiency.
  • Training and Evaluation Setup: CLIP-T5 experiments freeze CLIP and feed combined visual-text inputs to the side network while the backbone receives text inputs only.
  • Evaluation: GLUE reports task-specific metrics and maximum memory usage measured during training and evaluation on RTE.
  • Evaluation: VQA, GQA, and NLVR2 use accuracy, whereas MSCOCO uses CIDEr, with reported values averaged over three seeds.

5 Experimental Results

Across NLP and vision-and-language experiments, LST provides a stronger accuracy–memory trade-off than competing PETL methods, especially under limited GPU memory. Its advantages persist across model sizes, layer-dropping settings, initialization strategies, and comparisons with compression, side-tuning, and Y-tuning.

  • GLUE Results: 69% memory savings versus full fine-tuning and 2.7x more savings than Adapter and LoRA occur at similar parameter usage on GLUE.
  • GLUE Results: LST achieves higher accuracy than other PETL methods in the low-memory regime and scales to larger T5 models with strong GLUE performance.
  • GLUE Results: LST outperforms Y-tuning by a large margin with fewer updated parameters on GLUE tasks except STS-B.
  • Vision-and-Language Results: On vision-and-language tasks, LST is the only method that fits a similar-parameter configuration into a single 16GB GPU while remaining competitive with full fine-tuning and Adapter.
  • Accuracy–Memory Trade-off: LST remains robust across side-network sizes, whereas reducing Adapter hidden dimensions or LoRA ranks improves memory efficiency but significantly reduces performance.
  • Layer Efficiency: With layer dropping or freezing across memory budgets, LST has a better performance–memory trade-off and layer dropping generally reduces training cost without hurting performance.
  • Ablations: Pruned-network initialization generally helps performance across side-network sizes compared with random initialization.
  • Ablations: LST significantly outperforms network compression and Side-tuning, supporting the value of backbone information and intermediate shortcut connections.

6 Conclusion

LST adapts large backbone networks with lower training memory by avoiding backpropagation through the backbone. Experiments support its accuracy–memory efficiency across NLP and vision-and-language tasks.

  • LST avoids backpropagation through the backbone network, reducing training memory requirements relative to recent parameter-efficient training techniques.
  • LST enables adaptation of larger and more powerful backbone networks under limited memory constraints.
  • LST achieves a more efficient accuracy-memory trade-off than recent baselines and benefits from intermediate shortcut connections.
  • LST extends beyond NLP and achieves strong results on vision-and-language tasks.

A Comparison of different layer design in the ladder side network

The ladder side network uses Transformer blocks and a gating mechanism to fuse backbone information, while alternative side-network designs are evaluated on GLUE.

  • The selected side-network architecture uses Transformer blocks and a gating mechanism to fuse backbone information into the side network.
  • Adapter blocks with a two-layer bottleneck structure were also tested as an alternative side-network design.
  • Table 4 compares different side-network designs on GLUE using average accuracy over three random seeds.

B Hyper-parameters

The appendices provide the hyper-parameters for the NLP and vision-and-language experiments, with fixed batch sizes reported for each setting.

  • Hyper-parameters for the NLP experiments are provided in Table 5, while those for the vision-and-language experiments are provided in Table 6.
  • The NLP experiments use a batch size of 100 for all methods.
  • The vision-and-language experiments use a batch size of 300 for all methods.
Loading 2206.06522v2…