PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from models beyond fine-tuning

Hongling Zheng, Li Shen, Anke Tang, Yong Luo, Han Hu, Bo Du, Yonggang Wen, Dacheng Tao

arXiv:2310.08184v2cs.AIcs.LG

TL;DR

Limited access to training data and the costs of retraining motivate learning from foundation models. This survey reviews LFM approaches and their methods, then synthesizes their real-world challenges and future research directions.

  • Problem

    Training data can be difficult to obtain because of privacy and storage costs, while retraining increasingly large foundation models requires substantial computation.

    Source: §1 · p. 1

  • Method

    The survey systematically reviews learning from models across fine-tuning, distillation, reuse, meta-learning, and model editing.

    Source: §8 · p. 16

  • Results

    The survey examines how these approaches perform on real-world problems and identifies challenges including interpretability, security, and fairness.

    Source: §8 · p. 17

  • Takeaways & Limitations

    The authors present the survey as a comprehensive reference for researchers seeking to understand LFM approaches, their challenges, and future directions.

    Source: §8 · p. 17

    For model distillation, application conditions remain harsh, algorithms are not stable enough, studies mostly use small datasets, and unified benchmarks are lacking.

    Source: §3.4 · p. 10

Abstract

from arXiv · show

Foundation models (FM) have demonstrated remarkable performance across a wide range of tasks (especially in the fields of natural language processing and computer vision), primarily attributed to their ability to comprehend instructions and access extensive, high-quality data. This not only showcases their current effectiveness but also sets a promising trajectory towards the development of artificial general intelligence. Unfortunately, due to multiple constraints, the raw data of the model used for large model training are often inaccessible, so the use of end-to-end models for downstream tasks has become a new research trend, which we call Learn From Model (LFM) in this article. LFM focuses on the research, modification, and design of FM based on the model interface, so as to better understand the model structure and weights (in a black box environment), and to generalize the model to downstream tasks. The study of LFM techniques can be broadly categorized into five major areas: model tuning, model distillation, model reuse, meta learning and model editing. Each category encompasses a repertoire of methods and strategies that aim to enhance the capabilities and performance of FM. This paper gives a comprehensive review of the current methods based on FM from the perspective of LFM, in order to help readers better understand the current research status and ideas. To conclude, we summarize the survey by highlighting several critical areas for future exploration and addressing open issues that require further attention from the research community. The relevant papers we investigated in this article can be accessed at https://github.com/ruthless-man/Awesome-Learn-from-Model

1 INTRODUCTION

Learn From Model (LFM) studies foundation models to understand their behavior and improve or adapt them, especially when training data and retraining resources are difficult to obtain. The survey organizes this work into model-learning approaches and reviews their methods and future directions.

1 INTRODUCTION

  • LFM studies foundation models’ behavior, strengths, and shortcomings to optimize performance, identify problems, and support deployment.The survey frames LFM as research based on foundation models rather than the traditional learn-from-data paradigm.Source: §1 · pp. 1, 3
  • Privacy and commercial interests can restrict access to training data, while model-based study avoids direct contact with sensitive data and often works through API inputs and outputs.The survey also notes that data privacy and storage costs make it difficult for smaller organizations and individuals to obtain data for high-quality models.Source: §1 · p. 1
  • Using a base model with downstream-task tuning offers an alternative to retraining as foundation-model scale and computational requirements grow.The survey contrasts GPT-3’s 175B parameters with GPT-4’s 1.8T parameters and describes significant computational-cost savings from the alternative.Source: §1 · pp. 1–2
  • The survey categorizes foundation-model applications as tuning, distillation, fusion, reuse, meta-learning, and editing, and forecasts future applications.It describes tuning as adapting model parameters or components, distillation as transferring teacher knowledge to a smaller student, reuse as combining model predictions, and meta-learning as enabling faster adaptation.Source: §1 · p. 2
  • The authors propose the LFM concept and present a systematic survey with method classifications, application trends, and discussion of future opportunities and challenges.They intend the survey to support deeper understanding of the paradigm and serve as a stepping stone for research.Source: §1 · p. 3

2 MODEL TUNING

Model tuning adapts a foundation model for downstream tasks by changing parameters or adding components, with subtypes distinguished by where the modification occurs.

2 MODEL TUNING

  • Model tuning adapts foundation-model parameters for downstream tasks, using transfer learning to reduce data and training-cost burdens compared with retraining.The approach also uses common-sense knowledge stored in the foundation model to initialize parameters for new tasks.Source: §2 · p. 3
  • Successful tuning requires understanding the pretrained model’s internal structure and dynamics, including how inputs are encoded and which components influence predictions.Source: §2 · p. 3
  • Tuning is divided by modification location into weight tuning, input engineering, and database augmentation.These respectively modify internal parameters, design better input suggestions, and update model knowledge from external databases to support accuracy.Source: §2 · p. 3

2.1 Weight Tuning

Weight tuning adapts pretrained models by changing all parameters or by adding a smaller set of trainable adapter parameters. Adapter tuning reduces computational cost while retaining performance comparable to full-parameter fine-tuning, but added adapters can slow inference.

2.1.1 Fine-Tuning

  • Full-parameter fine-tuning updates all model parameters to adapt pretrained knowledge to specific tasks, but requires access to the model’s internal parameters and structure.The survey also identifies compute demands, overfitting with limited training data, catastrophic forgetting, and possible instability as limitations.Source: §2.1.1 · p. 4

2.1.2 Adapter Tuning

  • Adapter tuning keeps most pretrained parameters fixed and inserts trainable modules between model layers, using down- and up-projections with a residual connection.Adapters can be trained for multiple tasks and plugged into the pretrained model for new tasks.Source: §2.1.2 · pp. 4–5
  • Adapter tuning maintains performance comparable to full-parameter fine-tuning while reducing computational cost and the risk of overfitting.Adapters can also be shared or transferred among tasks, but added parameters may cause noticeable inference delay in deployment.Source: §2.1.2 · p. 5

2.2 Input Engineering

Input engineering adapts foundation models through prompts or supervised instruction-output examples. Prompt tuning addresses both open-weight and black-box settings, while instruction tuning trains models to follow instructions across tasks.

2.2.1 Prompt Tuning

  • Prompt tuning optimizes prompts while keeping the foundation model fixed, with approaches divided between white-box access to gradients and parameters and black-box access through queries.Black-box methods address cloud models whose parameters and gradients are unavailable to users.Source: §2.2.1 · p. 5
  • White-box prompt methods include trainable prefixes, soft prompts, anchor words, and virtual tokens, while black-box methods use derivative-free or policy-based prompt optimization.Other black-box approaches include emulator-based off-site tuning and reinforcement-learning prompt optimization [33].Source: §2.2.1 · pp. 5–6

2.2.2 Instruction Tuning

  • Instruction tuning supervises foundation models on instruction-output pairs, and prior work reports improved performance on tasks not seen during training [38], [71], [72].FLAN scales this approach to a 137B-parameter language model across more than 60 NLP datasets; T0 expands the tasks and prompts [73], [74].Source: §2.2.2 · p. 6
  • LLaVA uses instruction data generated by language-only GPT-4 to train a multimodal model connecting visual encoders and LLMs for visual and language understanding [38].The cited work presents this as an initial exploration of multimodal instruction tuning.Source: §2.2.2 · p. 6, Figure 4 · p. 6
  • Instruction tuning still calls for more instruction tasks and larger models, more precise and aligned training and inference templates, and balanced task data.The survey notes that each task need not have too many training samples.Source: §2.2.2 · p. 6

2.3 Database Augmentation

Database augmentation pairs a retriever that fetches external knowledge with a generator that uses it for predictions. The survey describes text retrieval augmentation and extensions to multimodal documents.

2.3 Database Augmentation

  • Database augmentation retrieves relevant external knowledge for a query and combines it with the query to make predictions, supporting access to up-to-date or domain-specific information.The survey also identifies potential benefits for reliability and handling domain-specific problems.Source: §2.3 · p. 6

2.3.1 Language Database Augmentation

  • REALM introduced retrieval augmentation as a whole concept for pretrained-model optimization, using a neural knowledge finder to access textual-corpus knowledge during pretraining, fine-tuning, and prediction.Later approaches include RAG, FID, Atlas, RETRO, and REPLUG, with differing retrieval and integration strategies, [85], [87],,.Source: §2.3.1 · p. 7, Figure 5 · p. 7

2.3.2 Multimodal Database Augmentation

  • Multimodal retrieval augmentation uses image-and-text documents; cited approaches include systems for multimodal memory, image generation, and simultaneous text-and-image retrieval and generation [89], [90],.RA-CM3 is described as retrieving and generating text and images from multimodal documents.Source: §2.3.2 · p. 7

2.4 Discussion And Application

The discussion frames LFM as requiring less data and computation than training from scratch, then highlights future tuning directions. It emphasizes reducing retrieval overhead and improving retrieval diversity.

2.4 Discussion And Application

  • Compared with training from scratch, LFM requires less data and computing power and can use a model’s existing general knowledge to address new tasks.The section distinguishes weight tuning, input engineering, and database augmentation as routes to model generalization.Source: §2.4 · p. 7
  • Future tuning work should explore self-supervised or unsupervised losses to reduce dependence on downstream task data.The discussion presents this as a direction for loss-function design.Source: §2.4 · p. 7
  • Larger retrieval memories may improve the chance of finding query-similar examples, but their retrieval overhead lowers inference efficiency, motivating memory-efficiency trade-offs.Data compression is offered as one possible approach.Source: §2.4 · p. 7
  • Tailored or multiple retrieval metrics could diversify retrieved results and broaden the useful information available for generation.The survey contrasts this direction with universal metrics that may limit retrieval diversity.Source: §2.4 · pp. 7–8

3 MODEL DISTILLATION

The section reviews data-free knowledge distillation from foundation models, organizing methods into noise optimization, generative reconstruction, and adversarial exploration. It also identifies practical constraints and research directions for these methods.

3.1 Noise Optimization

  • Noise optimization recovers semantically informative data by optimizing inputs against model-based regularizers, then aligns teacher and student predictions on the recovered data.BatchNorm feature statistics can provide sufficient average statistics, while the quality of recovered data affects student performance; noise optimization may require costly parameter updates and knowledge priors.Source: §3.1 · p. 8

3.2 Generative Reconstruction

  • Generative reconstruction uses a generator to synthesize data for distillation, with GAN-based methods using teacher guidance and regularizers to shape generated samples.Generated samples can suffer from within-class mode collapse, and their quality is seriously limited by generator structure.Source: §3.2 · pp. 8–9

3.3 Adversarial Exploration

  • Adversarial exploration alternates between generating samples that widen teacher–student disagreement and training the student to reduce that discrepancy.Generated samples may be unreliable, leaving some irrelevant examples less effective for knowledge transfer; prior approaches also search for poorly predicted samples and accelerate synthesis through reusable features.Source: §3.3 · pp. 9–10

3.4 Discussion And Application

  • The discussion proposes combining data-free distillation with downstream learning methods while identifying unstable algorithms, small-dataset evaluations, and the lack of unified benchmarks as unresolved constraints.The authors call for studying how foundation-model internal knowledge representations affect distillation performance and for designing distillation structures accordingly.Source: §3.4 · p. 10

4 MODEL REUSE

Model reuse combines knowledge from multiple trained models to retain useful information without always running an ensemble. The section surveys ensembles, model fusion approaches, and reuse goals including lower computation and improved complementarity.

4 MODEL REUSE

  • Model reuse addresses the trade-off between selecting one trained model, which discards others, and ensembling, which adds inference-time computation.Fusion aims to combine useful knowledge from individual models into a unified model, avoiding ensemble computational costs.Source: §4 · p. 10, §4.3 · p. 11

4.1 Ensemble Methods

  • Ensembles combine weighted predictions from multiple models, while efficient variants such as BatchEnsemble reduce computational and memory costs using shared weights and member-specific rank-one matrices.Tangent-space ensembles can reduce inference time from O(n) to O(1).Source: §4.1 · p. 10

4.2 Model Fusion

  • Model fusion seeks to extract and combine useful information from individual models into a more powerful model, addressing the inference cost of running multiple ensemble members.Weight interpolation combines models with the same network structure by combining their weights.Source: §4.2 · p. 10, §4.2.1 · p. 10

4.2.1 Weight Interpolation

  • Weight interpolation combines model parameters, including uniform averaging and model-specific weighted averaging, while task-vector addition or negation can manipulate performance on a target task.Uniform averaging is also called model soups or adapter soups; the cited discussion associates it with generalization, accuracy, robustness to distribution shifts, and out-of-distribution capability.Source: §4.2.1 · p. 11

4.2.2 Mode Connectivity Based Method

  • Mode connectivity describes paths between distinct local minima along which training or validation performance remains nearly constant, and model merging methods can align or combine models across tasks.Git Re-Basin permutes units to align models before merging, while ZipIt supports merging models trained on disjoint tasks.Source: §4.2.2 · p. 11

4.2.3 Straightforward Optimization

  • Straightforward optimization methods seek fused model weights or parameters by minimizing an objective, with approaches including neuron alignment and distance minimization.The section notes both closed-form and gradient-based optimization approaches.Source: §4.2.3 · p. 11

4.3 Discussion And Application

  • The discussion identifies model complementarity, lower computation than ensembling, knowledge extraction, robustness, and efficient training or editing as model-reuse goals.The survey lists applications including transfer learning, compression, continual learning, domain adaptation, personalization, efficient search, and multi-task learning.Source: §4.3 · pp. 11–12

5 META LEARNING

Meta-learning uses experience across related tasks to improve learning on new tasks, and data-free meta-learning extends this approach to collections of pretrained models without access to their training data. The section reviews white-box and black-box methods and identifies limitations that remain for applying them to foundation models.

5 META LEARNING

  • Data-free meta-learning (DFML) generalizes to new tasks by meta-learning from pretrained models without training data, in white-box or black-box settings.Black-box DFML is suited to settings where models are available only through APIs and their internal structure is unknown.Source: §5 · p. 12

5.1 White-box Data-free Meta-learning

  • MAML learns an initialization from source tasks so a target task can be learned with few training samples.The section also notes that MAML applies to classification, regression, and reinforcement learning.Source: §5.1 · p. 12, Figure 7 · p. 12
  • White-box methods transfer pretrained model parameters, but many MAML variants require original training data, constraining their use for large-model migration.CDCS adapts CodeBERT [186] with MAML for domain-specific languages and performs significantly better than directly fine-tuning the pretrained code model.Source: §5.1 · pp. 12–13

5.2 Black-box Data-free Meta-learning

  • Black-box DFML remains difficult because commercial models may expose only APIs, while some methods assume known parameters or shared model architectures.Wang predicts meta-initializations with a meta-trained black-box network using task embeddings.Source: §5.2 · p. 13
  • Black-box approaches include distilling meta-knowledge across APIs and synthesizing pseudo episodes to address limitations of model merging and scale.BiDf-MKD [188] uses API queries and zero-step estimation; PURER proposes a unified framework for synthesizing pseudo episodes.Source: §5.2 · p. 13

5.2.1 Discussion And Application

  • The section identifies needs for faster black-box DFML, more general white-box methods, lower computational cost, and stronger few-shot performance, which remains below fine-tuning.It also describes model interfaces as training resources for learning new tasks without large labeled datasets.Source: §5.2.1 · p. 13

6 MODEL EDITING

Model editing updates a pretrained model’s knowledge while aiming to preserve its outputs on unrelated inputs. The survey distinguishes memory-based and parameter-based strategies and discusses their methods and remaining challenges.

6 MODEL EDITING

  • Model editing changes knowledge in a specific domain while aiming not to adversely affect outputs for other inputs.Unlike fine-tuning, it targets the model’s overall or domain knowledge base rather than a specific downstream task.Source: §6 · p. 14

6.1 Memory Based Model Editing

  • Memory-based editors add independent components or stored edits while leaving the base model parameters unchanged.SERAC [21] stores edit descriptors in a cache and uses a range classifier and counterfactual model, but depends heavily on editing datasets to train them.Source: §6.1 · p. 14

6.2 Parameter Based Model Editing

  • Parameter-based editing directly updates model parameters and includes constrained tuning, locate-and-edit, and meta-learning approaches.These approaches use updates to alter model behavior rather than adding only an external memory.Source: §6.2 · p. 14, §6 · p. 14

6.2.1 Constrained Tuning

  • Constrained tuning fine-tunes on modified facts while constraining parameter changes to reduce interference with unmodified facts.The constraint is approximated using local continuity of the loss around the original parameters.Source: §6.2.1 · p. 14

6.2.2 Locate And Edit

  • Locate-and-edit methods identify knowledge-related neurons or mediating weights for targeted factual updates, while task vectors can be combined to control model behavior.MEMIT [219] can be expanded to store thousands of memories.Source: §6.2.2 · p. 15

6.2.3 Meta Learning

  • Meta-learning editors predict parameter updates for new edits, but direct parameter modification can cause collapse when editing large amounts of knowledge.MEND uses low-rank gradient transformations to apply local edits from a single desired input/output pair.Source: §6.2.3 · p. 15

6.3 Discussion And Application

  • Open challenges include editing non-factual knowledge, editing models accessible only through APIs, and evaluating strategies with broader benchmarks.Existing evaluations often sample unrelated knowledge from the same distribution as target knowledge.Source: §6.3 · p. 15

7 CHALLENGES AND FUTURE PROSPECT

The survey describes trends in foundation models, including multimodal systems and movement toward more general models, alongside open challenges in data quality, model internals, and safety. It calls for future work on these issues and on the demands of human annotation in safety training.

7.1 Summary and Research Trends

  • The authors call for interdisciplinary research and view foundation-model-based approaches as a promising area for addressing problems across research fields.The timeline in Figure 9 presents prominent models and their contributions over recent years.Source: §7.1 · pp. 15–16
  • The survey traces large-model progress and highlights multimodal systems as a way to combine language models’ reasoning with visual models’ perception.It also describes progression toward more general pretraining and notes reinforcement learning as a developing trend.Source: §7.1 · pp. 15–16, Figure 9 · p. 16

7.2 Open Research Questions And Future Prospect

  • Training-data imbalance can contribute to biased predictions, making data quality and representativeness a stated research priority.The section names data enhancement, rebalancing, and bias correction as possible ways to mitigate bias.Source: §7.2 · p. 16
  • Understanding how knowledge is represented inside large models is presented as important for reducing costs and applying models to downstream tasks more reasonably.The authors call for deeper study of model weights, algorithms, and internal mechanisms.Source: §7.2 · p. 16
  • Safety training with RLHF relies heavily on high-quality human annotations, so the authors call for reducing annotator burden and improving annotation efficiency and quality.The section identifies harmful, biased, or offensive outputs from manipulated models as a potential misuse risk.Source: §7.2 · p. 16

8 CONCLUSION

The conclusion presents LFM as a broad survey of ways to use pretrained models, covering methods, their advantages and limitations, and challenges in real-world implementation. It also discusses future research directions and issues including interpretability, security, and fairness.

8 CONCLUSION

  • The survey covers model fine-tuning, distillation, reuse, meta-learning, and editing, describing how each uses pretrained models in different contexts.It systematically reviews studies from early explorations to the present and discusses advantages and limitations.Source: §8 · pp. 16–17
  • The authors examine real-world challenges and discuss interpretability, security, fairness, and adapting models to new tasks or domains.They also suggest future directions for addressing these challenges.Source: §8 · p. 17

Related papers

Loading 2310.08184v2…