PaperlayerView exact revision on arXiv

Source-linked AI summary

Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs

Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew L. Leavitt, Naomi Saphra

arXiv:2309.07311v6cs.CL

TL;DR

Interpretability studies often examine only fully trained models, leaving training trajectories as a complementary lens on how model behavior develops. This paper tracks and manipulates syntactic attention structure in MLMs, finding that its abrupt emergence precedes grammatical capabilities and that it is necessary for handling complex linguistic phenomena.

  • Problem

    Most interpretability research examines fully trained models, while developmental analysis of training trajectories offers a complementary way to study how behavior emerges.

    Source: §1 · p. 1

  • Method

    The paper tracks SAS through MLM training and manipulates attention to syntactic neighbors with a regularizer to examine its role in grammatical capabilities.

    Source: §1 · p. 1, §2.2 · p. 3

  • Results

    SAS spikes during a precipitous loss drop before grammatical capabilities emerge, is necessary for handling complex linguistic phenomena, and briefly suppressing it improves model quality.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The developmental analysis uses SAS to illuminate simplicity bias, model complexity dynamics, and risks of changing the optimizer during phase transitions.

    Source: §5 · p. 8

Abstract

from arXiv · show

Most interpretability research in NLP focuses on understanding the behavior and features of a fully trained model. However, certain insights into model behavior may only be accessible by observing the trajectory of the training process. We present a case study of syntax acquisition in masked language models (MLMs) that demonstrates how analyzing the evolution of interpretable artifacts throughout training deepens our understanding of emergent behavior. In particular, we study Syntactic Attention Structure (SAS), a naturally emerging property of MLMs wherein specific Transformer heads tend to focus on specific syntactic relations. We identify a brief window in pretraining when models abruptly acquire SAS, concurrent with a steep drop in loss. This breakthrough precipitates the subsequent acquisition of linguistic capabilities. We then examine the causal role of SAS by manipulating SAS during training, and demonstrate that SAS is necessary for the development of grammatical capabilities. We further find that SAS competes with other beneficial traits during training, and that briefly suppressing SAS improves model quality. These findings offer an interpretation of a real-world example of both simplicity bias and breakthrough training dynamics.

1 INTRODUCTION

The paper tracks syntactic attention structure (SAS) during training to study how interpretable artifacts relate to masked language model capabilities. It reports that SAS emerges abruptly during a steep loss drop, before grammatical capabilities, and uses SAS interventions to examine its role.

1 INTRODUCTION

  • SAS consists of specialized attention heads focusing on words’ syntactic neighbors, and emerges naturally during conventional MLM pretraining.The study manipulates SAS during training to examine its impact on MLM capabilities.Source: §1 · p. 1
  • Monitoring latent syntactic structure during training reveals a precipitous loss drop composed of multiple phase transitions relating to various linguistic abilities.At structure onset, SAS spikes; after the spike, a break in BLiMP score signals capabilities onset.Source: §1 · pp. 1–2
  • SAS is necessary for handling complex linguistic phenomena, and briefly suppressing it improves model quality and accelerates convergence.Suppressing SAS after the alternative strategy onset damages performance and blocks SAS long-term.Source: §1 · p. 2
  • Figure 1 tracks loss, attention-derived parse UAS, and BLiMP accuracy across three seeds, showing abrupt emergence of SAS before grammatical capabilities.The seeds differ in post-onset UAS stability, while SAS emerges within a brief window in all three runs.Source: Figure 1 · p. 2

2 METHODS

The methods measure syntactic attention structure, manipulate it through a loss regularizer, and identify training breakthroughs from changes in metric trajectories.

2.1 SYNTACTIC ATTENTION STRUCTURE

  • SAS names the tendency for attention heads to specialize in specific syntactic relations, measured with an implicit parse and unlabeled attachment score.The measurement follows Clark et al. (2019).Source: §2.1 · pp. 2–3

2.2 CONTROLLING SAS

  • A syntactic regularizer rewards or penalizes attention to syntactic neighbors by adding a scaled syntacticity score to the MLM loss.Negative λ promotes SAS and positive λ suppresses it.Source: §2.2 · p. 3

2.3 IDENTIFYING BREAKTHROUGHS

  • Breakthroughs are identified as the sharpest change in a metric trajectory over a chosen distance, approximating the point of maximum acceleration.When required measurements are unavailable, the method imputes metric values from available checkpoints.Source: §2.3 · p. 3

3 MODELS AND DATA

The study trains BERTBase models on BookCorpus and Wikipedia, then evaluates grammatical capabilities, downstream tasks, and attention-derived parses.

3.1 ARCHITECTURE AND TRAINING

  • BERTBase uses 12 layers of 768 dimensions and is trained with AdamW; analysis is limited to checkpoints through 300K steps for numerical stability across seeds.Training uses BookCorpus and English Wikipedia, with a May 2020 Wikipedia dump and silver-standard dependency parses for regularized runs.Source: §3.1 · p. 3

3.2 FINETUNING AND PROBING

  • Evaluation covers GLUE fine-tuning, BLiMP minimal-pair grammatical judgments, and SAS parsing measured by UAS on Penn Treebank documents.BLiMP uses MLM pseudo-log-likelihood to compare acceptable and unacceptable sentences.Source: §3.2 · p. 4

4 RESULTS

The results track SAS through training and intervene on its development to examine its relationship to grammatical capabilities. They identify linked training transitions and a trade-off between early SAS suppression and later model quality.

4 RESULTS

  • Prior evidence from trained models offered only weak support that SAS is essential for grammar, motivating analysis of model development and intervention during learning.The authors note that passive observation and test-time interventions do not establish whether a component is necessary during learning.Source: §4 · p. 4

4.1 THE SYNTAX ACQUISITION PHASE

  • The loss drop spans 20K–30K steps and comprises a SAS-linked structure onset followed by a BLiMP capabilities onset; these transitions persist with continuous metrics and alternative training axes.Between the onsets, functional complexity rises during an abrupt memorization phase, despite declining for most of training.Source: §4.1 · p. 5

4.2 CONTROLLING SAS

  • Suppressing SAS throughout training damages early and long-term performance and prevents linguistically complex capabilities, while promoting SAS improves early MLM performance but hurts later quality.An early loss inflection around 6K steps under suppression suggests a competing strategy that suppression may let the model learn earlier and more effectively.Source: §4.2 · pp. 5–6

4.3 EARLY-STAGE SAS REGULARIZATION

  • Brief early SAS suppression improves later training effectiveness; BERT(3k)SAS- outperforms BERTBase after both models pass their structure and capabilities onsets, though the advantage later becomes nonsignificant.Some multistage models also show more consistent SAS than BERTBase; the authors posit that associative patterns learned during suppression may support performance and later SAS acquisition.Source: §4.3 · pp. 6–7

4.3.1 WHEN CAN WE RECOVER THE SAS PHASE TRANSITION?

  • The longest SAS suppressions delay and attenuate the UAS spike until it lacks a clear inflection; changing the regularizer during the alternative-strategy transition yields the worst model quality.The authors report that preventing the SAS phase transition is accompanied by significantly lower MLM performance than BERTBase.Source: §4.3.1 · p. 7, Figure 4 · p. 7, Figure 5 · p. 8

5 DISCUSSION AND CONCLUSIONS

The discussion uses training history and interventions to argue that SAS is important to grammatical development, while also showing that interpretable syntactic structure can compete with other learning strategies. It connects these findings to phase transitions and cautions that interventions and training settings constrain their interpretation.

5.1 EARLY DYNAMICS AND SIMPLICITY BIAS

  • SAS can act as a disadvantageously strong prior: suppressing it leads to an alternative context strategy, while simpler early data may encourage simplistic strategies.The authors also compare early SAS suppression to pretraining and suggest that homogenizing early data in a curriculum could help early performance but harm later performance.Source: §5.1 · pp. 8–9

5.2 PHASE TRANSITIONS

  • Changing regularization during a phase transition produces substantially worse model quality across metrics, whereas releasing it before or after the transition causes less damage.Releasing suppression before the alternative strategy onset can recover the SAS transition; releasing it after that transition does not recover SAS but limits damage to quality metrics.Source: §5.2 · p. 9

5.3 INTERPRETABILITY EPISTEMOLOGY

  • The authors caution that attention patterns alone may not establish information flow, and that instance-level observations or interventions in trained models leave alternative explanations.Their developmental interventions support a role for SAS, but the intervention may also affect a closely associated structural pattern.Source: §5.3 · p. 9, §C.2.1 · pp. 19–20

5 DISCUSSION AND CONCLUSIONS

  • Developmental analysis and training interventions support that some grammatical capabilities depend on SAS, although models eventually find alternative representations for some capabilities.The authors argue that this training-history perspective adds evidence beyond observing or intervening on a single trained model.Source: §5 · p. 8, §5.3 · p. 9

A SYNTACTIC ATTENTION STRUCTURE PROBE

  • The syntactic attention probe predicts a word’s parent from each head’s attention, then selects the best-performing head for each dependency relation and aggregates accuracy as UAS.UAS is measured on 1,000 Wall Street Journal Penn Treebank documents using Stanford Dependencies as silver labels.Source: A SYNTACTIC ATTENTION STRUCTURE PROBE · p. 17

B SYNTACTIC REGULARIZER

  • The syntactic regularizer changes attention to syntactic neighbors by adding a scaled term to MLM loss; negative λ promotes SAS and positive λ suppresses it.Source: B SYNTACTIC REGULARIZER · p. 17

C RELATED WORK

  • The paper connects interpretability research with training-dynamics research, drawing on work about simplicity bias, phase changes, and interventions during learning.It distinguishes its positive effects from critical-learning-period studies: suppressed behaviors are learned once they are permitted.Source: C RELATED WORK · p. 18, §C.1 · p. 18, §C.2.1 · p. 19

C.1 SIMPLICITY BIAS AND PHASE CHANGES

  • Phase changes in representational complexity align with generalization performance, but the authors note that seed sensitivity and hyperparameter settings may affect whether breakthroughs occur.Source: §C.1 · p. 18

D BLIMP IMPLEMENTATION DETAILS

This appendix describes how BLiMP grammatical accuracy is calculated and reports that implicit parse accuracy does not significantly correlate with either MLM test loss or grammatical capabilities across 25 MultiBERT seeds.

D BLIMP IMPLEMENTATION DETAILS

  • PLL scores a sentence by summing each token’s log probability when that token is masked and conditioned on the remaining context.Source: D BLIMP IMPLEMENTATION DETAILS · p. 20
  • BLiMP accuracy is the proportion of minimal pairs in which the acceptable sentence receives a higher PLL, or lower PPPL, than its unacceptable alternative.BLiMP includes 67 challenges of 1,000 minimal pairs each, covering syntactic, semantic, and morphological phenomena.Source: D BLIMP IMPLEMENTATION DETAILS · p. 20

E CORRELATION BETWEEN UAS AND CAPABILITIES

F MULTIBERTS DEVELOPMENTAL ANALYSIS

MultiBERTs checkpoints align with the same developmental transition, while analyses of complexity, optimization scales, and continuous metrics test how broadly the pattern holds. The transitions persist across these checks, alongside a two-phase complexity trajectory during SAS acquisition.

F MULTIBERTS DEVELOPMENTAL ANALYSIS

  • MultiBERTs remains compatible with the same phase transition despite a slightly faster UAS plateau and somewhat better loss and BLiMP results than the reproduction.Its UAS shows an initial spike followed by a plateau within 20K steps; the BLiMP increase also appears within that window.Source: F MULTIBERTS DEVELOPMENTAL ANALYSIS · p. 21

G COMPLEXITY AND COMPRESSION

  • Complexity trends toward compression overall, but expands during SAS acquisition before declining at the onset of advanced capabilities.This pattern is framed as a memorization phase followed by a compression phase.Source: G COMPLEXITY AND COMPRESSION · pp. 21–22

H IS THE PHASE TRANSITION CAUSED BY ABRUPT CHANGES IN STEP SIZE?

I IS THE PHASE TRANSITION AN ARTIFACT OF THRESHOLDING?

  • The phase transition remains clear with continuous alternatives to UAS accuracy and BLiMP accuracy.The alternatives measure attention on the correct syntactic target and the relative probability assigned to the correct versus incorrect CLS token.Source: I IS THE PHASE TRANSITION AN ARTIFACT… · p. 22

J GLUE TASK ANALYSIS

GLUE task outcomes vary: most show an accuracy breakthrough at structure onset, while most show no marked improvement after brief early SAS suppression.

J GLUE TASK ANALYSIS

  • Most GLUE tasks show an accuracy breakthrough at structure onset, though not all do.Source: J GLUE TASK ANALYSIS · p. 22
  • Most GLUE tasks show no marked improvement after brief early SAS suppression, and stable tasks decline with continued suppression past alternative strategy onset.Source: J GLUE TASK ANALYSIS · p. 22

K BLIMP ANALYSIS

BLiMP responses vary across tasks and SAS-regularization schedules, while the complexity analysis describes its measures and focuses on intrinsic dimension. The reported BLiMP pattern includes changes at both structure and capabilities onset.

K BLIMP ANALYSIS

  • Most BLiMP tasks reach maximum accuracy when SAS regularization is released after brief suppression; release at alternative strategy onset produces a dip and is poorest except on ellipsis.Source: K BLIMP ANALYSIS · p. 23
  • In baseline BERTBase training, most BLiMP tasks improve at capabilities onset, while some decline at structure onset.Source: K BLIMP ANALYSIS · p. 23

L COMPLEXITY METRICS

L.1 INTRINSIC DIMENSION

  • Two-NN estimates intrinsic dimension from the growth of neighborhood counts with radius; the analysis applies it to BERTBase [CLS] embeddings using cosine similarity.The analysis also presents empirical Fisher and weight norm dynamics.Source: §L.1 · p. 24

L.2 OTHER COMPLEXITY METRICS

The other complexity metrics show distinct training dynamics: classifier weight magnitude rises, while Fisher information increases during SAS acquisition and declines after capabilities onset.

L.2 OTHER COMPLEXITY METRICS

  • Classifier weight magnitude rises throughout training, with an upward inflection at capabilities onset.Source: §L.2 · p. 24
  • Fisher Information rises sharply during SAS acquisition, then decreases slowly after capabilities onset as the paper describes a compression phase and improved generalization.Source: §L.2 · p. 24

M WHAT IS THE ALTERNATIVE STRATEGY?

The section characterizes the SAS-suppressed model’s alternative strategy as relying on long-range semantic context rather than local syntax, alongside gains in longer-context n-gram prediction. Compared with BERTBase, its context-length breakthroughs appear more sequential, although checkpoint timing limits conclusions about whether BERTBase’s transitions are simultaneous.

M WHAT IS THE ALTERNATIVE STRATEGY?

  • BERTSAS- shows a faster initial increase in n-gram modeling than BERTBase, with its loss break associated with gains in long-range context.As training continues under SAS suppression, performance spikes for increasingly smaller contexts over a short window.Source: M WHAT IS THE ALTERNATIVE STRATEGY? · pp. 25–26
  • BERTSAS- attention is less tied to target-word position, and its higher per-sample entropy suggests attention remains concentrated on a few tokens that need not be nearby.The authors suggest semantic factors may guide attention, but caution that attention is not itself an importance metric.Source: M WHAT IS THE ALTERNATIVE STRATEGY? · p. 26

M.1 ONE BIG BREAKTHROUGH OR MANY SMALL BREAKTHROUGHS?

  • BERTBase shows n-gram breakthroughs across context lengths during the structure onset, while BERTSAS- appears to progress from longer to more local contexts.The checkpoints are too widely spaced to determine whether BERTBase has consecutive phase changes or near-simultaneous ones.Source: M WHAT IS THE ALTERNATIVE STRATEGY? · p. 25, §M.1 · pp. 26, 29
  • Phrase-level phase transitions can coexist with a smooth i.i.d. validation-loss curve, which may obscure transitions under distribution shifts.The authors present this as suggestive evidence that smooth loss curves can elide multiple phase transitions.Source: §M.1 · p. 29

N EARLY-SUPPRESSION TRAINING CURVES

Briefly suppressing SAS early can recover and augment the structure-onset transition, while longer suppression weakens those benefits. When evaluation is fixed at 50K steps after regularizer removal, shorter initial stages show less loss improvement, but the overall patterns remain consistent with fixed-total-time results.

N EARLY-SUPPRESSION TRAINING CURVES

  • Brief early SAS suppression can recover and augment the structure-onset UAS spike and loss drop, while continued suppression weakens these benefits and the transition to SAS.The multistage approach accelerates and augments the structure onset and improves model quality during the first 100K steps.Source: N EARLY-SUPPRESSION TRAINING CURVES · p. 29

O CONTROLLING FOR TIME ALLOWED TO ACQUIRE SAS

  • At a fixed 50K steps after regularizer release, models with shorter initial stages improve loss less, while the overall patterns remain consistent with fixed-100K-step results.This comparison controls the time allowed for training after the regularizer is removed.Source: O CONTROLLING FOR TIME ALLOWED TO… · p. 29

P LONG TERM IMPACT OF MULTISTAGE SUPPRESSION

Longer-term results indicate that the early quality difference between BERT(3k) SAS- and BERTBase ceases to be statistically significant. Early suppression nevertheless accelerates convergence and leaves functional differences, while representational similarity later approaches that between BERTBase seeds.

P LONG TERM IMPACT OF MULTISTAGE SUPPRESSION

  • BERTBase and BERT(3k) SAS- output-function distance eventually stabilizes above the average between BERTBase seed pairs, suggesting their behaviors remain distinct absent another phase transition.The models initially become more similar in their output functions.Source: P LONG TERM IMPACT OF MULTISTAGE… · p. 29
  • CKA similarity diverges during the structure and capabilities onsets, then returns toward the BERTBase-seed average, even as output distributions may remain noticeably different.The comparison uses total variation distance for output distributions and centered kernel alignment for representations.Source: P LONG TERM IMPACT OF MULTISTAGE… · pp. 29, 31
  • Brief SAS suppression improves MLM loss, implicit parse accuracy, and linguistic capabilities early in training.The plotted multistage curves begin after the regularizer is released, so their displayed trajectories cover different starting times.Source: Figure 18 · p. 31, Figure 17 · p. 31
  • The quality difference between SAS- and BERTBase ceases to be statistically significant; avoiding SAS early accelerates convergence and leaves some functional differences in final models, while the critical learning period does not hold at scale.Source: P LONG TERM IMPACT OF MULTISTAGE… · p. 31

Related papers

Loading 2309.07311v6…