Source-linked AI summary
Model Reprogramming: Resource-Efficient Cross-Domain Machine Learning
Pin-Yu Chen
TL;DR
Resource-limited domains may lack data, suitable pretrained models, or the resources to develop and train them. This paper surveys model reprogramming, which reuses a frozen source model across domains by training added input and output layers. Reported use cases include new state-of-the-art accuracy on a challenging data-limited task and performance comparable to modern language models with fewer trainable parameters.
Problem
Resource-limited target domains may lack data, adequate pretrained models for finetuning, or resources for model development and training.
Source: §1 · p. 1Method
Model reprogramming repurposes a pretrained source model for target tasks without finetuning its parameters, training added input-transformation and output-mapping layers.
Source: §1 · p. 1Results
Reported applications include new state-of-the-art accuracy on a challenging data-limited task and WARP performance comparable to modern language models with fewer trainable parameters.
Source: §2.2 · pp. 4–5Takeaways & Limitations
Reprogramming can reuse a source model as an efficient feature extractor when target representations can be aligned with source representations.
Source: §3.1 · p. 6The framework assumes consistent data formats, target data dimension no greater than the source dimension, and no more target than source classes.
Source: §2.1 · p. 3
Abstract
from arXiv · showhide
In data-rich domains such as vision, language, and speech, deep learning prevails to deliver high-performance task-specific models and can even learn general task-agnostic representations for efficient finetuning to downstream tasks. However, deep learning in resource-limited domains still faces multiple challenges including (i) limited data, (ii) constrained model development cost, and (iii) lack of adequate pre-trained models for effective finetuning. This paper provides an overview of model reprogramming to bridge this gap. Model reprogramming enables resource-efficient cross-domain machine learning by repurposing and reusing a well-developed pre-trained model from a source domain to solve tasks in a target domain without model finetuning, where the source and target domains can be vastly different. In many applications, model reprogramming outperforms transfer learning and training from scratch. This paper elucidates the methodology of model reprogramming, summarizes existing use cases, provides a theoretical explanation of the success of model reprogramming, and concludes with a discussion on open-ended research questions and opportunities. A list of model reprogramming studies is actively maintained and updated at https://github.com/IBM/model-reprogramming.
1 Introduction
Deep learning’s resource demands make reuse attractive when target domains lack data, suitable pretrained models, or affordable training. The paper presents model reprogramming as a way to reuse a frozen source model across domains by training added input and output layers.
1 Introduction
- Resource-limited target domains may lack high-quality pretrained models, sufficient data, or the resources to develop and train models.Developing high-capacity models can also be costly; GPT-3 is cited as an example with an estimated training cost of about 4.6 million US dollars in 2020.Source: §1 · p. 1
- Model reprogramming repurposes a pretrained source-domain model for target tasks without finetuning its parameters, training added input-transformation and output-mapping layers instead.The paper frames this as resource-efficient cross-domain learning that reuses models developed in data-rich domains.Source: §1 · pp. 1–2
- Transfer learning varies in how many source-model parameters are finetuned, while training large models from scratch can perform poorly with small datasets and foundation models require massive pretraining data.Foundation models also use large models to learn general-purpose representations; model reprogramming is described as particularly suited to small-data settings.Source: Figure 1 · p. 1, §1 · p. 2
- The paper outlines a framework and use cases, theoretical explanations for reprogramming’s success, and open research questions and opportunities.It also identifies mathematical notation used in the paper.Source: §1 · p. 2
2 Model Reprogramming Framework
Model reprogramming repurposes a frozen source-domain model for target tasks by adding input transformation and output mapping layers. The section describes the method and surveys use cases across data formats.
2.1 Methodology of model reprogramming
- With consistent data formats, source and target domains need not be similar, but target input dimension and class count must not exceed the source's.The framework is presented for reprogramming a more complex source model to a target with lower input dimension and fewer classes.Source: §2.1 · p. 3
- The input transform maps target inputs into the source model's input format, using masked perturbations or zero-padding for continuous data and inserted tokens or embedding mappings for discrete data.For discrete tokens, the mapping can approximate target-token embeddings with source-token embeddings.Source: §2.1 · pp. 3–4
- Output mapping either averages scores over source labels assigned many-to-one to a target class or uses a trainable dense head; greedy mappings can outperform random assignments.Iterative greedy mapping and assigning more non-overlapping source labels are also reported to improve performance.Source: §2.1 · p. 4
- The source model remains frozen while input and output mapping parameters are optimized against target-task loss; gradient-free methods support black-box models when backpropagation is infeasible.The optimized added parameters are then used with the pretrained source model for evaluation.Source: Figure 2 · p. 3, §2.1 · p. 4
2.2 Model reprogramming use cases
- BAR reports new state-of-the-art ASD accuracy and reprograms commercial APIs for biomedical tasks at about US$20; V2S beats or ties the best baseline on 19 of 30 UCR datasets.The examples cover black-box biomedical applications and univariate time-series classification.Source: §2.2 · pp. 4–5
- R2DL improves over training from scratch in reduced-data biochemical sequence classification, while WARP is comparable to modern language models with fewer trainable parameters and demonstrates few-shot data efficiency.R2DL uses dictionary learning to align target-token embeddings with embeddings from a pretrained language model.Source: §2.2 · p. 5
3 Theoretical Characterization of Model Reprogramming
The section explains model reprogramming through representation alignment and summarizes other proposed interpretations of its success.
3.1 Error analysis in representation alignment
- Under one-to-one label mapping, target risk is bounded by source risk plus the representation-alignment error between source and reprogrammed target data.The alignment error is measured using Wasserstein-1 distance between their representations under the same source model.Source: §3.1 · pp. 5–6
- Lower source risk and better alignment, measured by smaller Wasserstein distance, are conditions associated with lower target risk.The analysis interprets reprogramming as using a pretrained source model as a feature extractor while learning representation alignment.Source: §3.1 · p. 6
3.2 Other interpretations
- First-order analysis links optimal loss decrement to the ℓ1-norm of average input gradients; success is associated with more aligned target gradients and higher input dimensionality.Other accounts connect language-model data efficiency to trainable input prompts, while vision-to-vision reprogramming reduces to visual prompting.Source: §3.2 · p. 6
4 Open-ended Research Questions, Ongoing Efforts, and Opportunities
The section identifies open questions around supervision, foundation models, model properties, joint tuning, and heterogeneous computing. It also describes early applications and possible system-level benefits.
Model Reprogramming beyond Supervision
- Most existing work focuses on supervised reprogramming, leaving the role of unlabeled data and supervision-free training as an open question.The section highlights semi-supervised, unsupervised, and self-supervised approaches, potentially with data augmentation.Source: §4 · p. 6
Reprogramming Foundation Models
- Resource limits can restrict access to foundation models, motivating study of reprogramming them as an affordable route to resource-efficient reuse.The section connects this direction to potential acceleration of AI-assisted scientific discovery and democratization of machine learning research.Source: §4 · p. 6
Reprogramming for Improved Model Properties
- An open direction is using reprogramming as an efficient calibration tool to improve trustworthiness properties while minimally affecting original utility.Examples include fairness, robustness, explainability, privacy, uncertainty quantification, and energy efficiency.Source: §4 · p. 6
Joint Model Reprogramming and Fine-tuning
- Joint reprogramming and fine-tuning may outperform standalone approaches, but choosing which pretrained parameters to tune is challenging in resource-limited settings.Careful joint tuning has improved performance in low-resource speech-command recognition and music-genre classification.Source: §4 · p. 6
Implications on Machine Learning Systems with Heterogeneous Computing
- Keeping the source model server-side while placing added layers at edge devices may reduce communication and local storage needs; reported privacy-utility gains over standard transfer learning cover centralized and federated settings.The section identifies edge, cloud, and federated systems as settings with heterogeneous computation and memory constraints.Source: §4 · pp. 6–7