Fine-tuning can cripple your foundation model; preserving features may be the solution

1 Introduction

• Continual fine-tuning matters when models must absorb previously unknown knowledge while retaining prior knowledge, since retraining from scratch is too expensive to be feasible.The paper highlights fields such as medicine, where information and treatments change over time.Source: §1 · p. 2
• LDIFS reduces concept forgetting while fine-tuning on downstream tasks, without affecting downstream-task performance.It minimizes feature-space distance between pre-trained and fine-tuned models, using features from different internal representations.Source: §1 · p. 2
• In a continual setup of three sequences of three tasks, LDIFS outperforms fine-tuning and classic continual-learning methods in minimizing forgetting without compromising task performance.The reported setup evaluates both preservation of pre-trained knowledge and performance on fine-tuned tasks.Source: §1 · pp. 2–3
• The paper benchmarks six end-to-end fine-tuning methods on ten downstream tasks and finds concept forgetting across all six.The authors describe this as the first thorough analysis and evaluation of concept forgetting for fine-tuning multimodal foundation models.Source: §1 · p. 2
• L2SP outperforms other fine-tuning baselines on concept forgetting, motivating analysis of how fine-tuning methods differ.The paper uses this analysis to motivate a feature-space alternative to parameter-space distance.Source: §1 · p. 2

2 A brief note on fine-tuning

• CLIP uses image and text encoders trained contrastively on 400 million image-text pairs; zero-shot prediction selects the class whose text encoding best matches the image encoding.The class names are represented as text encodings and used as the parameters of a linear classifier.Source: §2 · p. 3
• Linear probing trains an added linear layer on image-encoder features, while without training it reduces to zero-shot prediction.The layer is trained with a standard logistic-regression problem.Source: §2 · p. 3
• ZS-init-CE fine-tunes the image encoder and a zero-shot-initialized linear head with cross-entropy, whereas LP-init-CE initializes the head through linear probing first.Both methods then fine-tune the entire model end-to-end.Source: §2 · p. 3
• ZS-init-L2SP adds a penalty on the distance between pre-trained and fine-tuned image-encoder weights, while LP-init-L2SP uses a linearly probed head before applying the same loss.The weight-distance penalty is computed only between the pre-trained and fine-tuned image encoders.Source: §2 · pp. 3–4
• FLYP fine-tunes both CLIP encoders using contrastive loss, while FLYP-CE changes the loss to cross-entropy using cosine similarities as logits.Both methods fine-tune the image and text encoders end-to-end.Source: §2 · p. 4

3.1 Quantifying Concept Forgetting

• After image-encoder fine-tuning, linear-probe accuracy is a better indicator than zero-shot accuracy of whether image representations still recognize task concepts.Fine-tuning can disrupt alignment with the text encoder, while linear probing tests whether image representations remain linearly separable.Source: §3.1 · p. 4
• Concept forgetting is measured as the change in linear-probe accuracy between fine-tuned and pretrained encoders on a monitored dataset not used for fine-tuning.Negative ∆LP indicates forgetting; zero indicates knowledge accumulation, and positive values indicate knowledge gain or positive forward transfer.Source: §3.1 · pp. 4–5
• Concept forgetting differs from catastrophic forgetting because foundation-model pretraining knowledge is difficult to quantify directly, motivating probes on smaller relevant concept domains.The pretraining data may be inaccessible and span millions or billions of self-supervised examples, so the exact forgotten knowledge cannot be measured.Source: §3.1 · p. 5

3.2 Benchmarking concept forgetting

• The benchmark uses CLIP ViT-B/32 and six end-to-end fine-tuning methods across ten downstream image-classification tasks with varied semantic concepts.The study tracks linear-probe performance during fine-tuning.Source: §3 · p. 4, §3.2 · pp. 5–6
• On EuroSAT, all six fine-tuning methods reduce ∆LP on eight other datasets; DTD is an exception with LP-init-L2SP, where ∆LP is mostly positive before reaching zero.The rise in EuroSAT ∆LP accompanies negative ∆LP on the other datasets, indicating concept forgetting.Source: Figure 2 · p. 6, §3.2 · p. 6
• LP-init-L2SP has the lowest negative ∆LP on datasets other than EuroSAT, often ending near zero, but has relatively lower ∆LP on EuroSAT itself.It initially dips on other datasets before recovering later in fine-tuning.Source: Figure 3 · p. 6, §3.2 · p. 6
• Most fine-tuning methods diverge from pretrained parameters, while L2SP baselines are exceptions; downstream-task performance alone therefore misses an additional effect worth evaluating.The section argues that concept forgetting requires its own evaluation alongside the conventional fine-tuning-task criterion.Source: §3.2 · p. 7

• Feature-space distance is proposed as a better indicator of encoded-concept similarity than parameter-space distance, which need not capture input-output behavior.The authors note that distinct parameter sets can be far apart while producing similar outputs in a specific domain.Source: §4 · p. 7
• Methods that suffer substantial concept forgetting generally diverge in feature space, while both L2SP variants partly recover pretrained behavior late in fine-tuning.The pattern holds on fine-tuning train and test sets and on other datasets.Source: §4 · p. 7
• LDIFS penalizes feature-space distance on fine-tuning samples; EuroSAT-only regularization also keeps features close on CIFAR10, CIFAR100, and GTSRB.LDIFS targets feature-space rather than parameter-space distance, and the observed feature preservation extends beyond the fine-tuning dataset.Source: §4 · p. 8
• In 8 of 10 tasks, LP-init-LDIFS has the highest average ΔLP on other tasks while remaining competitive on downstream accuracy.Figure 8 reports the test-set ΔLP comparison for EuroSAT fine-tuning; Table 1 summarizes downstream accuracy and average ΔLP across 10 tasks.Source: §4 · p. 9, Figure 8 · p. 9, Table 1 · p. 9
• Why CLIP and DINOv2 outperform FLAVA and MAE remains unclear; pretraining-set size, self-supervised loss, or other factors may explain the difference.Source: §4 · p. 10
• Across CLIP, FLAVA, DINOv2, and MAE, LP-init-LDIFS has the highest, mostly positive ΔLP; larger ViT-L models forget less than ViT-B models.Classic fine-tuning baselines show negative ΔLP across the evaluated models and architectures.Source: §4 · p. 10, Table 2 · p. 10

• The experiments use three sequences of three tasks, and report performance on sequence tasks plus an average over six other datasets.Figure 9 plots ALP for two sequences and marks task switches with vertical red lines.Source: §5 · pp. 10–11, Figure 9 · p. 11
• Continual ΔLP compares the final model’s linear-probe performance with the best earlier model’s performance on a dataset, allowing for positive forward transfer.Source: §5 · p. 11
• Across all three sequences, LP-init-LDIFS minimizes forgetting on earlier and unused datasets while remaining competitive on accuracy and outperforming classic continual-learning baselines.It also consistently outperforms L2SP on accuracy on the final RESISC45 task.Source: §5 · p. 11

6 Related Work & Discussion

• Appendix Table 3 reports ΔLP and ALP% for three-task sequences, while Table 4 compares LDIFS with five classic continual-learning methods.Table 3’s caption specifies performance on fine-tuned tasks and averaged performance on six other datasets.Source: Table 3 · p. 12, Table 4 · p. 12
• End-to-end fine-tuning outperforms adapters and prompt tuning on nine image-classification tasks, while Wise-FT can further improve concept forgetting in end-to-end methods.When linear probing trails end-to-end fine-tuning substantially, adapters and prompt tuning do not bridge the gap well.Source: §6 · p. 12
• LDIFS directly regularizes the L2 distance between intermediate features, including shallower layers, unlike classic distillation from logits or last-layer features.This avoids adding layers to distill intermediate representations.Source: §6 · p. 12
• On EuroSAT, LDIFS with only last-layer features is less performant than LDIFS using the full concatenated feature vector.The paper identifies earlier features as crucial to LDIFS performance.Source: §6 · p. 13, Figure 10 · p. 13
• Unlike LPIPS, which measures perceptual similarity between images using a frozen model, LDIFS compares pretrained and fine-tuned model features for the same image.LDIFS aims to preserve the pretrained model’s input-output behavior.Source: §6 · p. 13

7 Conclusions & Remarks

• LDIFS significantly alleviates concept forgetting without impeding downstream performance across ten classification tasks and three sequences of three continual-fine-tuning tasks.The authors motivate LDIFS by observing that pretraining encodes concepts in feature space and that L2SP’s parameter-space proximity suffers less forgetting.Source: §7 · p. 13
• End-to-end fine-tuning can damage recognition of concepts outside the target task, which the paper calls concept forgetting.This is undesirable for foundation models trained to encode information about many real-world concepts.Source: §7 · p. 13
• The cross-validated λLDIFS may not yield the best model for either concept forgetting or downstream performance.The paper cross-validates it on a held-out set from the fine-tuning task.Source: §7 · p. 13
• LDIFS uses a fixed set of evenly spaced features; which features encode which concepts remains open, as do concept evaluation beyond task-based performance and study across model families.The authors also flag the risk of preserving biases or undesirable behaviors from the pretrained model.Source: §7 · p. 13

A Training details

• The study fine-tunes CLIP ViT-B/32 on ten image-classification datasets, using eight methods for 80 training runs and 712 evaluation runs.Runs use a single NVIDIA A100 GPU; models are evaluated at checkpoints every 20% of training completion.Source: A Training details · p. 18
• Training uses AdamW with initial learning rate 1e-5, weight decay 0.1, batch size 128, and a cosine learning-rate schedule with 500 warmup steps.Epoch counts vary by dataset, from 10 to 76, with a minimum of 10 epochs.Source: A Training details · p. 18
• The LDIFS coefficient λLDIFS is selected by validation search over seven values, with λLDIFS = 10 used for all reported results.The authors report that 10 produced the best performance over datasets in general.Source: A Training details · p. 18

B Computing Φθ(x)

• For each input, LDIFS concatenates normalized features from five intermediate ViT-B/32 points, each flattened to 38,400 dimensions, yielding a 192,000-dimensional vector.The five-point design limits the memory cost of storing features from every intermediate layer.Source: B Computing Φθ(x) · p. 18

C Additional results

• The section presents additional empirical results to supplement the paper’s observations and conclusions.The additional results include zero-shot accuracy and ΔZS alongside the other reported analyses.Source: C Additional results · p. 19

C.1 Concept forgetting with AZS and ΔZS

• The appendix reports ΔZS as the change in zero-shot accuracy from the pretrained to fine-tuned model; negative values indicate forgetting, while positive values indicate knowledge gain or positive forward transfer.The authors include this additional measure while noting that zero-shot linear classifiers may not indicate concept recognition after fine-tuning.Source: §C.1 · p. 19

C.2 Crippling effect of fine-tuning

• On their respective fine-tuned test sets, models broadly achieve high accuracy that increases during fine-tuning; LP-init baselines’ lower zero-shot accuracy does not translate into lower ALP.Both ZS- and LP-init L2SP baselines have relatively lower fine-tuned accuracy in zero-shot and ALP evaluations.Source: §C.2 · p. 19
• On eight datasets other than the fine-tuned task, zero-shot and linear-probe performance broadly decline during fine-tuning; LP-init-L2SP consistently shows higher accuracy over fine-tuning than other baselines.The comparisons cover EuroSAT-, GTSRB-, and SVHN-fine-tuned models.Source: §C.2 · p. 19

C.3 Analysing L2SP

• L2SP moves away from the pre-trained model before converging toward it in both parameter and feature space, unlike other baselines.The feature-space pattern holds on EuroSAT train, test, and other datasets.Source: §C.3 · p. 19

C.4 Performance of LDIFS

• LP-init-LDIFS minimizes concept forgetting while retaining competitive fine-tuned-task accuracy and consistently improving on L2SP.When it is not best on a dataset pair, it is often a close second.Source: §C.4 · p. 20

C.5 Comparison with Adapters and Prompt Tuning

• End-to-end fine-tuning achieves the best test accuracy across nine tasks; adapters and prompt tuning do not close the linear-probe gap on tasks such as SVHN.The authors argue that when the pre-trained encoder performs poorly on a downstream task, end-to-end fine-tuning is needed for state-of-the-art results.Source: §C.5 · p. 21

C.6 Comparison with Wise-FT

• Wise-FT consistently reduces concept forgetting across fine-tuning baselines, while their relative ordering remains unchanged and LDIFS stays superior.Wise-FT tunes its interpolation weight on a held-out downstream validation set.Source: §C.6 · p. 21

C.7 Additional results on continual fine-tuning

• The joint-training comparison measures ΔLP as the difference between final-sequence performance and the best performance previously achieved on each task.At each step, joint training retrains on the combination of earlier-task data and the new task’s data.Source: §C.7 · p. 21
• On tasks outside the sequence, LDIFS significantly outperforms joint training, while remaining competitive on the final task and slightly underperforming on earlier tasks.Joint training uses data from all earlier tasks along with the incoming task at each sequence step.Source: §C.7 · p. 21
• On Split ImageNet-R, LP-init-LDIFS outperforms the compared pre-trained continual-learning baselines.The authors note that LDIFS was not specifically targeted to these benchmarks.Source: §C.7 · p. 22

C.8 Last Layer Ablation

• Using only last-layer feature differences fails to minimize full feature-space distance, suggesting learned concepts can also reside in shallower network layers.The authors therefore identify distilling from those layers as crucial.Source: §C.8 · p. 22

C.9 Visualizing Table 1

• The plots place each dataset’s test LP accuracy on the x-axis and average test LP accuracy on the remaining datasets on the y-axis; LDIFS baselines consistently lie top-right.The top-right position indicates retained performance on other datasets alongside strong accuracy on the fine-tuned dataset.Source: §C.9 · p. 22, Figure 14 · p. 23
• For models evaluated on datasets other than their fine-tuning task, LP-init-LDIFS almost consistently beats the other baselines, including LP-init-L2SP, on transferability.The comparison covers models fine-tuned on EuroSAT, GTSRB, and SVHN.Source: Figure 16 · p. 24, Figure 20 · p. 25
• LP-init-LDIFS outperforms other baselines, including LP-init-L2SP, in minimizing concept forgetting and improving fine-tuned-task performance.The table reports ΔZS, ΔLP, and ALP for models fully fine-tuned on nine classification datasets.Source: Table 9 · p. 26