Linear Connectivity Reveals Generalization Strategies

1 INTRODUCTION

• Earlier mode-connectivity results relied exclusively on image classification, although methods assuming linear connectivity can fail outside that setting.Prior work presented connectivity as an inductive bias of SGD and linked it to in-domain generalization.Source: §1 · p. 1
• Diagnostic sets reveal that similarly finetuned models can vary in their generalization to out-of-distribution linguistic structures.Some models perform well in-domain while using brittle strategies that fail to incorporate position information.Source: §1 · p. 1
• The paper examines how loss-surface barriers relate to generalization strategies across NLI, paraphrase, and grammatical acceptability.The motivating gap is that prior work had not linked different generalization behaviors to loss-surface geometry.Source: §1 · p. 1
• Across NLI, paraphrase, and acceptability tasks, models with similar diagnostic-set behavior are linearly connected on in-domain loss, while models with different behavior tend to be disconnected.The study links these basins to distinct generalization strategies, including syntactic and lexical-overlap heuristics in NLI.Source: §1 · p. 2

2 IDENTIFYING GENERALIZATION STRATEGIES

• Models with poor diagnostic-set performance are called heuristic, while those with high performance are called generalizing; the study examines NLI, paraphrase, and grammaticality.Models are initialized from BERT and differ by random seed, which controls classifier-head initialization and data order.Source: §2 · p. 2

2.1 NATURAL LANGUAGE INFERENCE

• HANS tests NLI heuristics such as lexical overlap, subsequence, and constituent overlap; higher accuracy on non-entailing HANS-LO examples indicates less reliance on lexical overlap.The NLI models are evaluated on the MNLI matched validation set.Source: §2.1 · p. 2

2.2 PARAPHRASE

• PAWS-QQP uses permuted question pairs that can violate lexical-overlap heuristics to distinguish paraphrase generalization behavior.QQP is a binary duplicate-question task.Source: §2.2 · p. 3

3 LINEAR MODE CONNECTIVITY

• Heuristic models connect to one another, as do generalizing models, but barriers in in-domain loss separate the two groups.During interpolation, heuristic-model pairs can improve on HANS-LO loss, while low interpolated loss does not always yield higher accuracy.Source: §3 · p. 4
• A mixed plane through heuristic and generalizing models has a large central barrier, whereas same-group planes have smaller barriers.The barrier is smallest for a perimeter of generalizing models and largest for a mixed perimeter.Source: §3 · p. 4, Figure 2 · p. 4

3.1 CONVEX BASINS

• An ϵ-convex basin is a convex region where the loss obeys a relaxed Jensen inequality; trained models form one when their convex hull has that property.At ϵ = 0, the loss is convex within the region.Source: §3.1 · p. 5
• Pairwise non-increasing linear connections are necessary but not sufficient for every convex combination to have non-increasing loss.Within an ϵ-convex basin, any two points connect linearly with loss that decays monotonically within ϵ to shared minima.Source: §3.1 · p. 5

4 THE CONVEXITY GAP

• Sharpness is not a good predictor of the quality of a model’s generalization strategy.The paper argues against explaining barriers between different heuristics as an effect of sharper minima in heuristic models.Source: §4 · p. 5

4.1 CLUSTERING

• Basin membership predicts generalization behavior better than optimum width or ϵ-sharpness, and CG-based centroid distances correlate more strongly with HANS performance than sharpness.The convexity gap is used for spectral clustering of models based on distances on the in-domain loss surface.Source: §4 · pp. 5–6, §4.1 · p. 6
• The larger MNLI basin is syntax-aware and associated with constituent-overlap behavior, while the smaller basin is syntax-unaware and relies on unordered word sets.Cluster membership accounts for part of the heavy tail in HANS-LO performance, though in-domain performance also differs between clusters.Source: §4.1 · p. 6, Figure 5 · p. 7
• QQP models also form distinct linearly connected clusters, and cluster membership predicts performance on PAWS-QQP.The figure presents CG over QQP validation loss and sorts models by PAWS-QQP performance.Source: §4.1 · p. 7, Figure 6 · p. 7

4.2 GENERALIZATION BASINS TRAP TRAINING TRAJECTORIES

• Models central to a heuristic cluster become more firmly assigned to it later in training, supporting the view that basins can trap optimization trajectories.The findings also suggest that early training behavior may predict a model’s eventual heuristic.Source: §4.2 · p. 7

5 RELATED WORK

• Prior work found nonlinear paths connecting trained models, while other studies proposed neuron permutation to explain apparent linear disconnection.This work instead studies linear connectivity without permutation.Source: §5 · pp. 7–8
• During QQP finetuning, heuristic models near the cluster boundary move toward the generalizing cluster, while central models move toward their respective centroids.The clustering uses the CG metric on the QQP validation loss surface.Source: Figure 7 · p. 8
• Shared-pretrained-checkpoint connectivity and low-loss interpolation after pruning were reported in image classification, while related weight ensembling was less effective in text classification.The cited work on weight ensembling depends on assumptions of linear mode connectivity.Source: §5 · p. 8
• NLP diagnostic sets test behaviors such as compositionality and grammaticality, including responses to word-order perturbations.The paper uses rule-based diagnostic datasets to distinguish mechanical heuristics in text classifiers.Source: §5 · p. 8
• Earlier studies found diverse decision boundaries and OOD behaviors; this work links identically trained models to different OOD behaviors and interpretable strategies.The related work describes boundary diversity within low-loss volumes and diversity under OOD generalization.Source: §5 · p. 8

6 DISCUSSION AND FUTURE WORK

• Image-classification findings favor a single basin after neuron alignment, but the authors question generalizing this view beyond image classification.They point to differences in global structure and near-linearity early in training as possible reasons.Source: §6 · p. 9
• MNLI and QQP share a binary sentence-pair format, and CoLA outlier models are rare, limiting how broadly these three tasks establish basin-strategy alignment.The authors suggest more diverse tasks could test the consistency of these alignments.Source: §6 · p. 9
• Future work could measure how transfer-learning priors distribute models across basins and examine effects of pretraining length, data scheduling, architecture, and available features.The authors identify this as an area for theoretical work in optimization.Source: §6 · p. 9
• The authors propose testing whether training-loss CG can predict OOD generalization early, noting that models tend to remain in their cluster late in training.They raise predicting position use versus bag-of-words behavior as an example.Source: §6 · p. 9
• Different strategies may relate to CoLA bimodality and NLI diagnostic-set variance, and barriers may help explain why weight averaging can fail in text classifiers.The authors suggest distinguishing strategy basins could improve weight ensembling.Source: §6 · p. 9

7 CONCLUSIONS

• The paper concludes that pretrained models may favor a small set of basins, and that linear connectivity can indicate shared generalization strategies under domain shift across three text tasks.Neurons were aligned without permutation, though the authors note that permutations they did not find may exist.Source: §7 · p. 9

A LINGUISTIC ACCEPTABILITY

• CoLA’s OOD diagnostic sets come from different linguistics papers than its training and ID sets, testing transfer beyond the acceptability rules in the ID sources.CoLA consists of acceptable and unacceptable English sentences.Source: A LINGUISTIC ACCEPTABILITY · p. 15

A.1 EXPERIMENTAL DETAILS

• The CoLA analysis used HuggingFace defaults for six epochs, with learning rate 2 × 10−5, batch size 32, and no weight decay; the authors attribute the single outlier cluster possibly to chance.The script difference noted was the lack of different classification-head initializations between HuggingFace runs.Source: §A.1 · p. 15, §A.2 · p. 15

A.2 CLUSTERING

• One of 48 CoLA models accounted for all substantial interpolation convexity gaps and outperformed the others on OOD generalization.The authors interpret this as suggesting that models with different generalization behavior are disconnected; Figure 8 shows the CG heatmap and cluster membership against OOD performance.Source: §A.2 · p. 15, Figure 8 · p. 15

B THEORETICAL RESULT ON CONVEXITY GAPS

• An ϵ-basin guarantees CG(w1, w2) ≤ ϵ for every pair of models on its surface.The proof applies the basin’s defining inequality to pairs of interpolated models and bounds the supremum of their barrier heights.Source: B THEORETICAL RESULT ON CONVEXITY GAPS · p. 16

C HANS PERFORMANCE ON SUBSEQUENCE HEURISTICS

• Models performing poorly on subsequence heuristics tend to lie in the bag-of-words basin, though these clusters are less distinct than for constituent or lexical-overlap heuristics.The comparison is shown in Fig. 9, which sorts MNLI model pairs by HANS-subsequence performance and reports their convexity gaps.Source: C HANS PERFORMANCE ON SUBSEQUENCE… · p. 16, Figure 9 · p. 16

D FLEXIBLE MODE CONNECTIVITY

• Nonlinear paths between heuristic and generalizing models can maintain near-constant ID loss while HANS-LO generalization worsens along the path.This contrasts with linear interpolation, where heuristic-to-generalizing connections show peaks in ID loss but no corresponding OOD-loss spikes.Source: D FLEXIBLE MODE CONNECTIVITY · p. 17, Figure 10 · p. 17

E CORRELATIONS BETWEEN PERFORMANCE ON HANS SUBSETS

• HANS-LO non-entailing accuracy correlates positively with ID accuracy, while avoiding the constituent-overlap heuristic may instead reflect failure to acquire syntax.Subsequence performance correlates positively with lexical-overlap and constituent performance; for each heuristic, non-entailing and entailing subset performance correlate negatively.Source: E CORRELATIONS BETWEEN PERFORMANCE ON… · p. 17, Figure 11 · p. 18

F.1 FINETUNING THE MODELS

• QQP models use three epochs, a 2 × 10−5 learning rate, batch size 32, and 0.01 weight decay from bert-base-uncased.The passage identifies the Google script and AdamW with warm-up during the first 10% of training followed by a linear weight-decay schedule.Source: §F.1 · p. 18

F.2 INTERPOLATIONS

• Interpolation experiments evaluate 512 shared random target-dataset samples at α intervals of 0.1, then compute convexity gaps from interpolated-model losses.With 256 MNLI samples, the correlation coefficient in Fig. 3(a) fell to ρ = −0.65, but cluster assignments remained unchanged.Source: §F.2 · pp. 18–19

F.3 MODEL EVALUATIONS

• Model evaluations use full available validation and diagnostic data, and HANS labels are formed by mapping MNLI contradiction and neutral logits to non-entailment.The listed evaluation sets include HANS non-entailing test samples, PAWS-QQP dev and test splits, and the full QQP and MNLI validation sets.Source: §F.3 · p. 19

F.4 COMPUTING SHARPNESS

• Sharpness is computed by maximizing loss within a bounded parameter perturbation region; the reported experiments use 32,768 MNLI samples and ϵ = 1 × 10−5.The reported computational total is approximately 994 GPU-hours across a mix of NVIDIA RTX8000 and V100 nodes.Source: §F.4 · p. 19, §F.4.1 · p. 19

• Training-loss convexity gaps strongly relate to HANS-LO performance, even without validation or test data.The authors suggest future work could predict generalization early using training behavior alone.Source: G TRAINING LOSS · p. 19

• All 12 syntax-unaware NLI models are at least 1.3 standard deviations below the syntax-aware cluster mean on HANS-LO, but only 1 standard deviation below it on MNLI validation.The authors conclude that basin membership is more predictive of NLI OOD than ID behavior; constituent performance’s negative correlation with ID performance supports a syntax-awareness distinction.Source: H IN-DOMAIN GENERALIZATION · p. 20
• The single CoLA model outside the main basin is 2.6 standard deviations from the mean on OOD performance and 0.6 on ID performance.The authors describe this as stronger evidence that CoLA basin membership relates to OOD generalization over ID.Source: H IN-DOMAIN GENERALIZATION · p. 20
• On QQP, ID validation accuracy predicts convexity gap better than PAWS, while the authors leave better characterizations of basin strategies to future work.They also note a more linear relationship between ID accuracy and centroid distance than between OOD accuracy and centroid distance in both MNLI and QQP.Source: H IN-DOMAIN GENERALIZATION · p. 20, Figure 14 · p. 21
• ID performance and ID convexity gaps use the same validation set because test sets are not public, exposing ID performance during clustering.This caveat casts doubt on the alignment between clusters and ID performance.Source: H IN-DOMAIN GENERALIZATION · pp. 20–21

I ALTERNATIVE BARRIER MEASUREMENTS

• −0.49 is the Pearson correlation for NLI clustering with barrier height, far lower in magnitude than in convexity-gap metric space.The comparison is shown against HANS-LO accuracy.Source: I ALTERNATIVE BARRIER MEASUREMENTS · p. 21, Figure 15 · p. 22
• AUC gives the same Pearson correlation coefficient as barrier height, but its heatmap clusters are much less crisp.AUC is computed after subtracting the curve’s lowest point and measuring the area under the shifted curve.Source: I ALTERNATIVE BARRIER MEASUREMENTS · p. 21, Figure 16 · p. 22
• Euclidean distance has an extremely strong clustering effect and predicts generalization behavior on MNLI and QQP.The figures compare distance on the MNLI and QQP validation loss surfaces with HANS-LO and PAWS-QQP accuracy, respectively.Source: I ALTERNATIVE BARRIER MEASUREMENTS · p. 21, Figure 17 · p. 22, Figure 18 · p. 23

J ROBERTA RESULTS

• RoBERTa generalizes significantly better on HANS-LO than BERT but shows no clear clustering, suggesting basin selection may leave room for benefits from longer pretraining.Figure 19 reports the RoBERTa MNLI clustering results.Source: J ROBERTA RESULTS · p. 23, Figure 19 · p. 23

K RELATION TO FINETUNING DYNAMICS

• LP-FT reduces convexity gaps and strong multi-cluster behavior, slightly shifts HANS-LO behavior, and may limit ensembling benefits through greater model homogeneity.LP-FT first trains the linear classifier head in isolation, then fine-tunes the rest of the network.Source: K RELATION TO FINETUNING DYNAMICS · pp. 23–24, Figure 20 · p. 24

L THE RELATIONSHIP BETWEEN EUCLIDEAN DISTANCE AND CG

• Euclidean distance is more responsive to barrier size within clusters than between them, so within-basin placement may affect distance differently.The comparison covers MNLI and QQP connections.Source: L THE RELATIONSHIP BETWEEN EUCLIDEAN… · pp. 24–25, Figure 21 · p. 24