PaperlayerView exact revision on arXiv

Source-linked AI summary

What Happens To BERT Embeddings During Fine-tuning?

Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, Ian Tenney

arXiv:2004.14448v1cs.CL

TL;DR

The paper asks how much BERT’s linguistic knowledge and internal representations change during downstream fine-tuning. Using probing classifiers, RSA, and model ablations, it finds that linguistic features remain available while changes concentrate mainly in the top layers, with task-dependent depth. Fine-tuning strongly alters in-domain representations but has a weaker effect out of domain, indicating a scope for improving generalization and transfer.

  • Problem

    Comparatively little is understood about how BERT’s representation space changes during fine-tuning, including whether linguistic features are preserved, where updates occur, and whether they generalize.

  • Method

    The paper combines supervised probing classifiers, Representational Similarity Analysis, and truncation and partial-freezing ablations to analyze fine-tuned BERT representations.

  • Results

    Fine-tuning preserves linguistic structures, primarily changes the top few layers with task-dependent depth, and affects in-domain representations more than out-of-domain representations.

  • Takeaways & Limitations

    Fine-tuning is a conservative, largely in-domain transfer process that preserves linguistic features while leaving room to improve generalization and transferability.

  • Takeaways & Limitations

    The standard fine-tuning recipe affects only a fraction of model capacity and remains specific to in-domain examples.

Abstract

from arXiv · show

While there has been much recent work studying how linguistic information is encoded in pre-trained sentence representations, comparatively little is understood about how these models change when adapted to solve downstream tasks. Using a suite of analysis techniques (probing classifiers, Representational Similarity Analysis, and model ablations), we investigate how fine-tuning affects the representations of the BERT model. We find that while fine-tuning necessarily makes significant changes, it does not lead to catastrophic forgetting of linguistic phenomena. We instead find that fine-tuning primarily affects the top layers of BERT, but with noteworthy variation across tasks. In particular, dependency parsing reconfigures most of the model, whereas SQuAD and MNLI appear to involve much shallower processing. Finally, we also find that fine-tuning has a weaker effect on representations of out-of-domain sentences, suggesting room for improvement in model generalization.

1 Introduction

This paper examines how BERT’s internal representations change during fine-tuning, asking whether linguistic features are preserved, where updates occur, and whether changes generalize across domains. It uses probing, RSA, and ablations to compare fine-tuned models with the pretrained encoder.

  • 1 Introduction: Fine-tuning is studied as a change in BERT’s representation space across dependency parsing, MNLI, and SQuAD.The analysis targets linguistic-feature encoding, layer localization, and domain generalization.
  • 1 Introduction: Three complementary techniques—probing classifiers, RSA, and model ablations—measure linguistic information, activation changes, and end-task effects.The ablations include truncation and partial freezing.
  • 1 Introduction: Fine-tuning does not eliminate linguistic features, but its effects vary substantially across tasks.The paper contrasts shallow changes for SQuAD and MNLI with deeper changes for dependency parsing.
  • 1 Introduction: Fine-tuning changes in-domain representations more strongly than representations of out-of-domain sentences.Out-of-domain representations remain similar to those of the pretrained encoder.

2 Related Work

Earlier work established that pretrained encoders contain broad linguistic information, while comparatively little work examined how fine-tuning changes those representations. Existing studies also reported strong benchmark performance alongside limited out-of-domain generalization and possible reliance on heuristics.

  • 2 Related Work: Probing studies found that pretrained encoders represent phenomena ranging from syntactic properties to entities and coreference.These studies use supervised diagnostic classifiers to test information in model activations.
  • 2 Related Work: Unsupervised analyses examined how Transformer representations evolve across layers, contexts, and training.Methods included PW-CCA and Representational Similarity Analysis.
  • 2 Related Work: Fine-tuned encoders achieved strong benchmark performance, but challenge-set studies found limited generalization to out-of-domain data.Prior work therefore raised questions about whether downstream models rely on shallow heuristics.
  • 2 Related Work: Prior representation-focused studies examined task-specific processing or optimization, but offered limited comparisons with the pretrained encoder.The paper identifies Gauthier and Levy’s RSA and structural-probe work as closest in approach.

3 Experimental Setup

The experiments compare the 12-layer BERT Base uncased model before and after fine-tuning on MNLI, SQuAD, and dependency parsing. Each task uses its corresponding prediction head and reported development or evaluation performance.

  • 3 Experimental Setup: The study uses the 12-layer BERT Base uncased variant, pretrained with masked language modeling and next sentence prediction.The pretrained model is called Base, while fine-tuned versions are named by task.
  • 3 Experimental Setup: MNLI contains over 433K sentence pairs, and BERT predicts textual entailment from the final-layer [CLS] representation.The model’s evaluation accuracy is 83.3 ± 0.1 across three trials.
  • 3 Experimental Setup: SQuADv1.1 contains over 100,000 question-answer pairs, with an output layer predicting answer start and end tokens.The average F1 score is 89.2 ± 0.2.
  • 3 Experimental Setup: Dependency parsing is included as a contrasting task because prior work suggested much required information already exists in pretrained BERT.The model achieves a development-set LAS of 96.3 ± 0.1.

4 What happens to linguistic features?

Fine-tuning largely preserves BERT’s encoded linguistic and syntactic information, although its effects vary across tasks and representation layers. Probing results show small changes for MNLI and SQuAD, improved syntactic performance for dependency parsing, and no catastrophic forgetting.

  • 4 What happens to linguistic features?: The probes test whether fine-tuned BERT retains linguistic features by comparing edge- and structural-probe performance with the pretrained model.Edge probes assess labeling information, while structural probes assess syntactic structure encoded in token representations.
  • 4 What happens to linguistic features?: Fine-tuned models retain most linguistic concepts discovered during unsupervised pre-training, with edge-probe F1 changes generally small relative to baseline differences.Run-to-run variation is ±0.7 for fine-tuning from the same checkpoint, compared with ±0.5 between probing runs and ±1.0 between pretraining runs.
  • 4 What happens to linguistic features?: Across edge-probing tasks, performance usually drops 0.5-2%, while MNLI and SQuAD show 1.5-3% syntactic declines and dependency parsing gains 4% on constituent labeling.Dependency parsing simultaneously loses performance on coreference, SPR, and relation classification tasks.
  • 4 What happens to linguistic features?: Dependency-parsing fine-tuning improves structural-probe correlation, root accuracy, and UUAS as early as layer 5, consistent with deeper changes to syntax representations.The probing metrics improve because the fine-tuning task is aligned with syntactic structure.
  • 4 What happens to linguistic features?: MNLI and SQuAD produce small structural-probe drops, especially in the final layer, while Spearman correlations change less than discrete root-accuracy and UUAS metrics.The results suggest that syntactic information remains largely preserved within the model’s syntactic subspace.
  • 4 What happens to linguistic features?: Together, the probing results suggest no catastrophic forgetting: linguistic information may not guide final predictions but remains available in the representations.This conclusion contrasts with evidence that fine-tuned models can rely on annotation artifacts, simple pattern matching, or other shortcuts.

5 What changes in the representations?

Fine-tuning changes BERT representations mainly in upper layers, but the depth and task relevance of those changes vary substantially: MNLI and SQuAD are shallower than dependency parsing.

  • Analysis methods: RSA compares representation spaces through pairwise similarity matrices and Pearson correlation, while ablations test whether changes matter for task performance.The study combines probing, RSA, partial freezing, and truncation to assess broad representational change and its behavioral relevance.
  • Representational Similarity Analysis: Across tasks, representational changes generally arise in BERT’s top layers, with little change near the input.The authors note optimization as one possible explanation for this layer pattern.
  • Representational Similarity Analysis: MNLI shows the smallest change, retaining similarity of 0.84 ± 0.02 at the second-to-last attention layer.The comparison uses Wikipedia sentences as inputs.
  • Representational Similarity Analysis: SQuAD remains similar to BERT Base through layer 7 before diverging steeply, indicating a deeper but still relatively shallow reconfiguration.Its changes are concentrated after layer 7 rather than throughout the encoder.
  • Layer ablations: Partial freezing and truncation corroborate shallow processing for MNLI and SQuAD, whereas dependency parsing performance drops more rapidly when fewer layers can adapt.MNLI remains effective with three attention layers, while SQuAD degrades when three or fewer layers are allowed to change.
  • Generalization: Fine-tuning changes token representations more strongly for in-domain than out-of-domain data.The comparison covers MNLI and SQuAD models tested on task-domain examples versus Wikipedia sentences.

6 Out-of-Domain Behavior

The paper tests whether fine-tuning changes BERT representations only for task-domain inputs or more broadly, using RSA comparisons across in-domain and Wikipedia sentences. Fine-tuned models diverge more from BERT Base on in-domain examples, while remaining more similar on Wikipedia sentences.

  • Method: RSA compares fine-tuned models with BERT Base across in-domain MNLI and SQuAD inputs and out-of-domain Wikipedia sentences.The analysis uses random development-set samples for the task domains and Wikipedia sentences as an out-of-domain control.
  • Results: Fine-tuned models diverge from BERT Base in the top layers, with substantially larger representational changes for in-domain examples.This trend holds for both MNLI and SQuAD fine-tuned models.
  • Results: Wikipedia sentences retain much higher similarity to BERT Base representations than in-domain examples.Wikipedia is used because it resembles the pre-training data.
  • Interpretation: These results suggest that fine-tuning changes representations for the fine-tuning domain while preserving more Base-like behavior elsewhere.The conclusion is based on the contrast between in-domain and Wikipedia RSA scores.
  • Robustness: The study also finds similar trends when using single-sentence MNLI and SQuAD inputs instead of premise-question pairs.The premise-only and question-only tests produced trends similar to those in Figure 5.

7 Conclusions

The paper concludes that fine-tuning preserves linguistic features while changing mainly the upper layers of BERT, with the depth of change varying substantially by task. Its effects are also stronger for in-domain than out-of-domain inputs, leaving room to improve transfer and generalization.

  • Linguistic representations: Supervised probing finds that linguistic structures learned during pretraining remain available after fine-tuning.The authors relate this preservation to the possibility that task models may still rely on shallow heuristics despite retaining linguistic features.
  • Layer changes: RSA and layer ablations show that fine-tuning changes only a fraction of BERT's capacity, concentrated in the top few layers with task-dependent variation.The conclusion is supported by analyses of representational similarity and model ablations.
  • Domain specificity: Fine-tuning changes in-domain representations more strongly than out-of-domain representations, which remain closer to the pre-trained model.This pattern suggests that fine-tuning is specific to task-domain examples rather than uniformly reshaping behavior.
  • Implications: The authors characterize current fine-tuning as conservative and identify potential to use more model capacity to improve generalization and transferability.The proposed opportunity follows from preserved linguistic features, limited layer changes, and domain-specific effects.
Loading 2004.14448v1…