Exploring Mode Connectivity for Pre-trained Language Models

• The study addresses a gap in understanding how high-performing minima reached under different adaptation configurations are geometrically related, using low-loss paths as its mode-connectivity measure.Directly probing the loss landscape is described as intractable at PLM scale.Source: §1 · p. 1
• Among training data order, tunable-parameter initialization, and training steps, initialization has the greatest effect on mode connectivity; fine-tuning connects better than parameter-efficient delta tuning such as adapters.Source: §1 · p. 2
• For minima trained on different datasets, connectivity is good when the datasets share a distribution without overlapping instances, while different-task minima are inherently disconnected and pre-training gradually brings their optimal regions closer.The authors interpret same-distribution connectivity as evidence of shared task-level knowledge rather than memorized examples, and suggest the pre-training pattern may help explain cross-task transferability.Source: §1 · p. 2
• Along paths from source-task to target-task minima, PLMs forget source-task examples and acquire target-task knowledge, prioritizing elusive source knowledge for forgetting and easy-to-grasp target knowledge for acquisition.The reported path analysis describes catastrophic forgetting on the source task and gradual absorption of target-task knowledge.Source: §1 · p. 2

• PLM adaptation strategies include hyperparameter search, pre-finetuning, prompt learning, and delta tuning, which updates only a few parameters while freezing the others (Ding et al., 2022).The passage also describes pre-finetuning as auxiliary-task training before target-task fine-tuning (Aghajanyan et al., 2021), and prompt learning as casting downstream tasks in the form of pre-training objectives (Brown et al., 2020; Schick and Schütze, 2021a,b).Source: §2 · p. 2
• Prior neural-network studies find low-loss paths between minima, often nonlinear; linear mode connectivity is a stronger same-basin condition associated with minima from different data orders and the same initialization (Frankle et al., 2020).Nonlinear connectivity has been studied by Garipov et al. (2018), Freeman and Bruna (2017), and Draxler et al. (2018).Source: §2 · pp. 2–3
• Mode-connectivity work has focused on non-pretrained vision models, with linear connectivity observed in pretrained vision models, while systematic study of PLMs remained lacking.Source: §2 · p. 3
• This work uses mode connectivity to study minima reached under different training configurations and extends prior same-dataset analyses to different datasets and knowledge variation along the path.Source: §2 · pp. 2–3

• Two configuration sets, C1 and C2, independently train PLM copies; the tuning method determines which parameters θ0 are adapted into each resulting minimum.Source: §3 · p. 3
• The evaluation first tests linear mode connectivity, motivated by the same-initialization result and the view that linearity indicates a closer connection than nonlinear paths.The path connects the two trained minima through interpolated parameter values.Source: §3 · p. 3
• Two minima count as mode connected when evenly distributed path interpolations show no significant loss barrier or performance drop.Source: §3 · p. 3

Effects of hyperparameters

• Different training orders preserve linear connectivity, while longer training can worsen adapter connectivity; fine-tuning remains well connected across training steps.The training-order experiment finds only a slight, negligible midpoint drop for adapter tuning.Source: Figure 1 · p. 4, §4 · p. 4

Effects of initialization and tuning method

• Fine-tuning generally connects better than adapter tuning, while different initializations especially disrupt adapter tuning’s linear connectivity.The authors hypothesize that tuning more parameters may improve connectivity, but leave that question for future work.Source: Figure 2 · p. 4, §4 · pp. 4–5

Non-linear mode connectivity

• A task-trained quadratic Bezier path connects adapter-tuning minima from different initializations without a significant performance drop.The endpoints are frozen while the curve parameters are optimized to minimize expected task loss along the path.Source: Figure 3 · p. 5, §4 · p. 5

Effects of data overlap

• Minima trained on disjoint MNLI samples remain well connected under both tuning methods, indicating connectivity reflects shared task-level knowledge rather than memorized instances.The experiment splits MNLI into two equal, non-overlapping training sets.Source: Figure 4 · p. 5, §4 · pp. 5–6

Effects of data domain

• Across MNLI and ANLI, interpolation shifts performance from the source task toward the target without a valley below both endpoints.The authors relate the opposite performance trends to source-task forgetting and gradual absorption of target-task knowledge.Source: Figure 5 · p. 6, §4 · p. 6

Pre-training and mode connectivity

• As pre-training increases, linear connectivity improves for both tuning methods; fine-tuning shows no performance drop after 6.25k pre-training steps.The authors conclude that good connectivity is acquired early in pre-training.Source: Figure 6 · p. 6, §4 · pp. 6–7

Pre-training and task boundaries

• Longer pre-training widens the overlap of MNLI and SST-2 high-performance regions; at 60k steps, neither task has a jointly poor region.Minima trained on different tasks are disconnected, but their optimal regions move closer with longer pre-training.Source: Figure 7 · p. 7, §4 · p. 7

Task knowledge along the path

• Along MNLI-to-ANLI interpolation, difficult source samples are forgotten first, while easy target knowledge is learned before elusive target knowledge.The analysis uses newly forgotten and newly memorized training samples and characterizes them through confidence and variability.Source: §4 · pp. 7–8, Figure 8 · p. 8

Weight Averaging

• Weight averaging selects the midpoint of a linear path, while a non-linear curve can be better connected in some cases.The authors suggest that linear interpolation may therefore miss an optimal model combination, and that training configurations can guide model choice for ensembling.Source: §5 · p. 8

Task-level Transferability

• Pre-training’s closer task boundaries may help explain PLMs’ cross-task transferability by making it easier to traverse between task regions without a loss barrier.The discussion presents this as a possible explanation, not a demonstrated cause of transferability.Source: §5 · p. 8

Knowledge Quantification

• Training-data memorization is only a rough estimate of task knowledge; the authors identify knowledge probing as a more granular avenue to explore.They connect path-wise knowledge analysis and quantification to research on knowledge distillation and transfer.Source: §5 · p. 8

6 Conclusions

• The analyses investigate how downstream adaptation configurations and pre-training affect PLM mode connectivity.They also examine knowledge variation along paths connecting different minima.Source: §6 · p. 8
• The paper examines task-knowledge variation along paths connecting different minima as part of its empirical analysis.This complements its analysis of adaptation configurations and pre-training.Source: §6 · p. 8
• The authors conclude that exploring PLM mode connectivity contributes to understanding the inner workings of downstream adaptation.They expect the evaluation setup and analyses to inspire further exploration.Source: §6 · p. 9

Limitations

• The analysis connects only two minima at a time, leaving simultaneous connections among multiple minima for future work.The authors identify more advanced tools for analyzing multiple minima as an open direction.Source: Limitations · p. 9
• The paper sets no strict threshold for how much performance drop makes minima not well-connected.The authors say their experimental results were significant enough that they did not consider a strict definition necessary.Source: Limitations · p. 9

A Details for Finding a Bezier Curve

• The curve-finding procedure optimizes only the Bezier control point to reduce expected task loss along the curve.The procedure uses a uniform sample of α in [0, 1] as a computationally tractable alternative, and initializes the control point at the endpoint midpoint for more stable training.Source: A Details for Finding a Bezier Curve · p. 13
• Initializing the control point at the midpoint of the endpoints makes curve training more stable than random-weight or pretrained-weight initialization.This initialization starts from a linear curve.Source: A Details for Finding a Bezier Curve · p. 13

B.1 Effects of the Learning Rate and the Batch Size

• Fine-tuning minima were always linearly connected in the tested settings, while adapter connectivity was sometimes poor and sometimes worsened with more training steps.The experiments varied learning rates and batch sizes for T5BASE on MNLI and SST-2; adapter results align with the paper’s finding that fine-tuning is generally better connected than delta tuning.Source: §B.1 · p. 13

B.2 Additional Experiments for the Effects of Training Steps

• The unequal-step experiments use independently initialized T5BASE endpoints and evaluate their linear interpolations on MNLI and ReCoRD.The 50k-step endpoint is compared with endpoints trained for 10k, 20k, 30k, or 40k steps.Source: §B.2 · p. 13
• Fine-tuning remains generally well connected across unequal training steps, but adapter connectivity worsens as the gap between endpoints grows.The comparison uses one endpoint trained for 50k steps and another trained for 10k, 20k, 30k, or 40k steps.Source: §B.2 · p. 13, Figure 11 · p. 15
• Additional linear-interpolation experiments vary learning rate and batch size for MNLI and SST-2 under fine-tuning and adapter tuning.The figures compare learning-rate pairs of 1 × 10−4 with 5 × 10−5 or 5 × 10−4, and batch-size pairs of 16 with 32 or 8.Source: Figure 9 · p. 14, §B.2 · p. 14, Figure 10 · p. 14
• Endpoint Euclidean distance increases with training steps, which may partially explain poorer mode connectivity.The distance analysis uses checkpoints from MNLI adaptation with fine-tuning and adapter tuning.Source: §B.2 · p. 14, Figure 12 · p. 15

B.3 More Experiments for Mode Connectivity along a Non-linear Path

• Across tested settings, nonlinear curves show no significant performance drop, supporting good connectivity between minima under those conditions.The experiments include adapter tuning with different data orders or non-overlapping samples from one distribution, and fine-tuning with different steps or initialization.Source: §B.3 · p. 15, Figure 13 · p. 16, Figure 14 · p. 16

B.4 Additional Experiments for the Effects of Data Overlap

• Additional data-overlap visualizations use ReCoRD and SST-2, complementing the main-paper MNLI experiment.Other settings are kept the same as in the main paper.Source: §B.4 · p. 15, Figure 15 · p. 17

B.5 Additional Experiments for the Effects of Data Domain

• Adapter-tuning visualizations extend the data-domain experiments beyond the main paper’s fine-tuning setting.Figure 16 compares minima trained on different data distributions of the same task.Source: §B.5 · p. 15, Figure 16 · p. 17

B.6 Additional Experiments for the Change of Mode Connectivity during Pre-training

• MNLI and QQP provide an additional task-pair visualization of connectivity changes during pre-training; only two task pairs are reported because of energy costs.Pilot studies found the conclusions consistent across diverse tasks.Source: §B.6 · p. 16, Figure 17 · p. 17

B.7 Performance along the Connecting Path

• A linear interpolation between independently trained MNLI weights outperforms both endpoints, and its selected solution performs better than the selected curved-path solution.The best development-set point among 24 evenly spaced curve points is evaluated on the test set; results average three runs.Source: §B.7 · p. 17, Table 1 · p. 18

B.8 Other Strategies for Weight Ensemble

• Matrix-wise parameter combination performs best among the three strategies and exceeds linear interpolation on MNLI.The compared strategies are layer-wise, module-wise, and matrix-wise division.Source: §B.8 · p. 18, Table 2 · p. 18

C Training Details

• Training details cover T5BASE checkpoints and AdamW, dataset splitting, and task-specific fine-tuning settings; adapter tuning uses a shared default learning rate and batch size.Datasets lacking test sets are repartitioned approximately 8:1:1; adapter defaults are 5 × 10−4 and batch size 16.Source: C Training Details · p. 18

C.1 Additional Details for the Effects of Initialization; C.2 Additional Details for Curve Finding; C.3 Additional Details for Calculating Confidence and Variability

• Initialization uses Gaussian noise or newly initialized modules; curve finding evaluates candidate interpolations, while confidence and variability use training dynamics.Curve finding evaluates α values 0.25, 0.5, and 0.75; MNLI and ANLI use 8 and 20 epochs, respectively, for confidence and variability.Source: §C.1 · p. 18, §C.2 · p. 19, §C.3 · p. 19

C.4 Additional Details for Pre-training RoBERTaBASE

• Pre-training uses concatenated Wikipedia and BookCorpus data with batch size 2048.The setup otherwise follows Liu et al. (2019b).Source: §C.4 · p. 19
• The RoBERTaBASE architecture is pre-trained with Adam on eight NVIDIA V100 GPUs using linear learning-rate decay.The stated optimizer settings are ϵ = 1 × 10−6, β1 = 0.9, and β2 = 0.98, with dropout 0.1 and weight decay 0.01.Source: §C.4 · p. 19
• A loss visualization reports MNLI linear interpolations between differently initialized minima at different pre-training steps.Figure 19 corresponds to the performance visualization in Figure 6.Source: Figure 19 · p. 19

D The Visualization of Loss for Interpolations

• Loss and performance trends are generally highly correlated, so the main paper reports performance while the appendix visualizes loss for most experiments.The loss figures cover data order, initialization, in-distribution data, pre-training steps, learning rate, and batch size.Source: D The Visualization of Loss for… · p. 19
• Loss interpolations compare minima trained with different data orders and different initializations.The corresponding performance visualizations are Figures 1 and 2.Source: Figure 20 · p. 20, Figure 21 · p. 20
• Further loss plots cover in-distribution minima, MNLI/SST-2 and MNLI/QQP task pairs across pre-training steps, and varied learning rates or batch sizes.The related performance visualizations are Figures 4, 7, 9, 10, 15, and 17.Source: Figure 22 · p. 20, Figure 23 · p. 20, Figure 24 · p. 21