Offsite-Tuning: Transfer Learning without Full Model

1 Introduction

• Fine-tuning is difficult because proprietary models can require users to share labeled data, while full-model tuning demands resources beyond most users’ reach.The passage cites GPT-3’s 175 billion parameters and 350GB GPU-memory requirement for storing parameters and inference.Source: §1 · pp. 1–2
• The model owner sends an adapter and emulator for data-owner tuning, then integrates the returned adapter into the full model.The adapter encodes task-specific knowledge; the compressed emulator mimics the rest of the model and supplies approximate gradients.Source: §1 · p. 2, Figure 2 · p. 2
• The approach avoids sharing either party’s protected asset and is more resource-efficient than tuning methods that require full-model weights.The data owner keeps training data private, and the model owner retains the full weights; the emulator is lossy-compressed.Source: §1 · p. 2
• Comparable downstream results to full-model fine-tuning come with up to 6.6× speedup and 5.6× memory reduction.Offsite-Tuning was evaluated on language and vision models; it also enables tuning OPT-6.7B and BLOOM-7.1B on a single GPU where this was previously not achievable.Source: §1 · p. 2

2 Related Work

• Foundation models are pretrained neural networks used for specific tasks, but fine-tuning and deploying them can be resource-intensive and may require sharing data with model owners.The passage notes that many such models are non-public.Source: §2 · p. 2
• Parameter-efficient fine-tuning updates or adds few parameters, but still requires the entire model’s weights and at least one full-model copy on-device.Adapter-tuning, Prefix-tuning, LoRA, and BitFit are examples of these methods.Source: §2 · pp. 2–3
• Federated learning keeps user data local, but gives each user the full model and assumes users can train its weights.The passage notes this is hardly feasible for large foundation models.Source: §2 · p. 3
• Decoupled learning divides end-to-end optimization into smaller subproblems, but has not been extensively explored for fine-tuning pretrained large foundation models.The described methods include auxiliary variables, delayed gradient descent, and model assembly.Source: §2 · p. 3

3 Problem Definition

• The setting requires keeping labeled training data from the model owner and the full model from the data owner.The goal is to tune on the data owner’s data without access to full model weights.Source: §3 · p. 3
• Direct fine-tuning optimizes a parameter update Δ by minimizing loss on the downstream dataset: Δ = arg minδ L(Θ + δ, D).The passage describes the model as changing from M_Θ to M_(Θ+Δ).Source: §3 · p. 3
• The substitute emulator M* must be significantly smaller and weaker than the foundation model, and users optimize it on the downstream dataset.The substitute’s tuned weights are denoted Δ* in the following objective.Source: §3 · p. 3
• The target is for plugging the substitute’s trained update into the original model to achieve performance similar to direct full-model fine-tuning.This is framed as the desired outcome without giving the data owner access to the original model.Source: §3 · p. 3
• Zero-shot performance measures the pretrained model on a downstream task without fine-tuning, while emulator performance measures the tuned substitute model.The passage denotes these as M_Θ and M* respectively.Source: §3 · p. 3
• Plug-in performance measures the original pretrained model with the substitute’s trained weights inserted, while full fine-tuning performance directly tunes the foundation model.The passage denotes these as M_(Θ+Δ*) and M_(Θ+Δ), respectively.Source: §3 · p. 3
• The stated effectiveness criteria require zero-shot performance below plug-in performance, emulator performance below plug-in performance, and plug-in performance approximately matching full fine-tuning.These comparisons express improvement from tuning, the original model’s edge over the emulator, and limited performance sacrifice for privacy.Source: §3 · p. 3

4.1 Framework Overview

• The model is split into a trainable adapter and frozen component; users tune the adapter with a lossy emulator, then return it for integration into the original model.Compression is intended to keep emulator performance below an acceptable level while improving efficiency and protecting model ownership.Source: §4.1 · p. 4

4.2 Adapter Selection

• The sandwich adapter includes both shallow and deep layers, and outperforms tuning only the last several layers across downstream tasks.The design is M = A1 ◦ E ◦ A2.Source: §4.2 · p. 4

4.3 Emulator Compression

• Layer-drop compression uniformly removes frozen-component layers, retains the first and last, and provides the best balance among emulator guidance, privacy, and size.When resources are available, knowledge distillation on the pre-training dataset uses the original component as supervision and MSE loss; proper distillation maintains similar transfer accuracy while preserving an accuracy gap.Source: §4.3 · p. 4

4.4 Connection to Parameter-Efficient Fine-tuning

• Offsite-Tuning is orthogonal to parameter-efficient fine-tuning: those methods reduce trainable parameters with full weights, while this framework targets privacy and efficiency without full-weight access.Small trainable modules from parameter-efficient methods can be incorporated into the adapter.Source: §4.4 · p. 5

5.1 Setup

• The evaluation covers GPT-2, OPT, and BLOOM on WikiText and eight question-answering benchmarks, plus vision models on six image-classification datasets.Language modeling uses perplexity on WikiText and accuracy on the other language benchmarks; vision performance is measured by accuracy.Source: §5.1 · p. 5

5.2 Results of Language Models

• Offsite-tuning improves zero-shot performance across all Table 2 tasks without full-model access, with an emulator-to-plug-in performance gap.For large models, plug-in performance is significantly better than zero-shot performance, while emulator performance remains noticeably lower.Source: Table 2 · p. 6, §5.2 · p. 6
• For medium-sized language models, plug-in performance is comparable to full-model fine-tuning, while emulator performance is significantly lower.These models have fewer than 2 billion parameters; the emulator is distilled, and Table 1 reports the results.Source: §5.2 · pp. 5–6

5.3 Results of Vision Models

• For CLIP and EVA, plug-in performance is comparable to full fine-tuning, while emulator performance is slightly lower.The evaluation uses ViT-G backbones with 1 billion parameters and distills the emulator on ImageNet for one epoch.Source: §5.3 · p. 6, Table 3 · p. 6
• At the same trainable-layer budget, tuning both the top and bottom language-model layers significantly outperforms tuning only the top or only the bottom.The ablation varies the number and position of adapter layers.Source: Figure 3 · p. 7
• Layer-drop achieves the best plug-in performance among the compression methods while maintaining a gap between plug-in and emulator performance.The caption also reports that layer-drop reduces emulator size while preserving plug-in performance.Source: Figure 4 · p. 7

5.4.1 POSITION AND NUMBER OF ADAPTER LAYERS

• With the same trainable-layer budget, tuning both top and bottom layers significantly outperformed tuning only the top or bottom layers.Increasing the number of tunable layers did not significantly reduce the gap between partial and full fine-tuning; the experiments use the top and bottom two layers as the adapter.Source: §5.4.1 · p. 7

5.4.3 EFFECT OF EMULATOR DISTILLATION

• Emulator distillation improves plug-in performance by 2.47 on OPT-1.3B and 4.75 on GPT-2-XL.The reported scores are WikiText-2 validation perplexities, where lower is better.Source: Table 4 · p. 7, §5.4.3 · p. 7

5.4.2 COMPRESSION METHODS FOR THE EMULATOR

• Layer-drop yielded the best plug-in performance and a clear emulator-to-plug-in gap, unlike the nonsignificant gap with other compression methods.The emulator retains two bottom and two top layers as the adapter while compressing the middle layers.Source: §5.4.2 · p. 7

5.5 Combining with Parameter-Efficient Fine-tuning

• Adapter-tuning and LoRA significantly reduce trainable parameters while maintaining plug-in performance.The comparisons use OPT-1.3B and GPT-2-XL on WikiText-2.Source: §5.5 · p. 7, Table 5 · p. 8
• Offsite-tuning combines with parameter-efficient fine-tuning by applying those methods only to the adapter layers.The tested methods were Adapter-tuning, LoRA, and BitFit.Source: §5.5 · p. 7
• BitFit failed to adapt the model, with a significant gap between plug-in and full-model fine-tuning performance.Source: §5.5 · p. 8

5.6 Efficiency

• 6.5x speedup and 5.6x reduction in memory usage were achieved by combining offsite-tuning with LoRA.Source: §5.6 · p. 8
• Offsite-tuning reduces both trainable parameters and the total parameters placed on-device during fine-tuning.Source: §5.6 · p. 8
• The reduced on-device parameter load increases fine-tuning throughput and reduces memory footprint.Source: §5.6 · p. 8

6 Discussion

• The authors describe personalizing language models on edge devices and adapting models to sensitive records without sharing those records with the model owner.They also note that not sharing the full model with the data owner protects model-owner privacy.Source: §6 · p. 8
• The work addresses data privacy and efficiency during adaptation, not privacy concerns during inference.Source: §6 · p. 8
• Using about one-third of a model’s middle layers as an emulator remains costly in scale for GPT-3, and compute-intensive distillation may be cost-prohibitive for larger models.The authors also say leakage has not been fully demonstrated absent and call for reconstruction studies and a clearer theoretical foundation.Source: §6 · p. 8

• Offsite-Tuning adapts foundation models to downstream tasks without access to full model parameters.Source: §7 · p. 8
• The framework is effective on billion-parameter language and vision foundation models.Source: §7 · p. 8
• Offsite-Tuning enables efficient foundation-model customization while addressing data and model privacy.Source: §7 · p. 8