PaperlayerView exact revision on arXiv

Source-linked AI summary

Mode Connectivity Beyond Classifiers: Evidence from Generative and Contrastive Models

Chengzheyi Yao, Yongzhao Zhang, Yongding Tian

arXiv:2608.30366v1cs.LG

TL;DR

Mode connectivity has been studied mainly in classifiers, leaving open whether independently trained generative and contrastive models share this property. The paper proposes an architecture-aware connection-building algorithm and reports continuous low-loss paths for DDPM and NanoCLIP, while noting limitations for distant modes and for generation quality.

  • Problem

    Existing empirical mode-connectivity evidence is largely confined to classifier-based models, leaving generative and contrastive architectures insufficiently studied.

  • Method

    The paper combines dataflow-based layer movement with Adam-based refinement in an architecture-aware two-stage connection-building pipeline.

  • Results

    The method constructs continuous low-loss paths between independently trained DDPM modes on Flowers102 and NanoCLIP modes on Flickr30k.

  • Takeaways & Limitations

    The results provide empirical evidence that mode connectivity extends to representative generative and contrastive models.

  • Takeaways & Limitations

    The refinement frequently fails to restore low loss when checkpoint separation is large, and intermediate DDPM checkpoints can have higher FID despite similar training loss.

Abstract

from arXiv · show

The loss landscape of Deep Neural Networks (DNNs) exhibits highly complex and non-convex properties. Recent studies have revealed the phenomenon of mode connectivity, demonstrating that independently trained network modes can be connected via a continuous low-loss path. However, existing mode connectivity research is predominantly confined to classifier-based models, leaving it an open question whether similar geometric properties exist in modern complex models. In this paper, we extend the boundaries of mode connectivity to generative and contrastive domains (specifically DDPM and NanoCLIP). Addressing the unique architecture of DDPM and CLIP, we propose an architecture-aware connection building algorithm. Extensive empirical results demonstrate for the first time that we successfully discover mode connectivity between independently trained DDPM and NanoCLIP modes. Our work provides a novel perspective for understanding the geometric properties of the loss landscapes in modern generative and contrastive models.

1 Introduction

Mode connectivity research has largely focused on classifier models, leaving its presence in modern generative and contrastive architectures unresolved. This paper addresses that gap with an architecture-aware method and reports successful connectivity for DDPM and NanoCLIP.

  • Prior empirical studies mainly examine unimodal supervised classifiers on small-scale CIFAR datasets, leaving generative and contrastive mode connectivity largely unexplored.
  • The paper extends mode connectivity to DDPM on Flowers102 and NanoCLIP on Flickr30K.
  • DDPM’s U-Net skip connections and NanoCLIP’s cross-modal image-text alignment create architectural challenges absent from conventional classifiers.
  • The proposed two-stage pipeline combines dataflow-based layer movement with optimizer selection during training refinement.The study uses Adam rather than the SGD commonly used by previous methods.
  • The work reports empirical evidence that independently trained DDPM and NanoCLIP modes can be connected through continuous low-loss paths.

2 Related Work

Mode connectivity research draws on theoretical explanations and empirical path-finding methods, but its empirical scope has remained centered on classifier models. This paper identifies generative and contrastive connectivity as the unresolved extension.

  • Theoretical Perspective: Singular learning theory explains low-loss subspaces through neural-network parameter degeneracies but does not provide algorithms for connecting specified modes.
  • Empirical Studies: AutoNEB constructs low-loss paths by inserting and optimizing intermediate pivots, while later methods address broader mode-pair connectivity empirically.
  • Empirical Studies: Previous empirical studies span multiple architectures and datasets, as summarized in Table 1, but remain confined to classifier-based models.
  • Linear Mode Connectivity: Linear mode connectivity differs from general mode connectivity because it often requires shared early optimization or symmetry-based parameter alignment.
  • Scope: This work investigates connectivity specifically between independently trained modes rather than modes sharing early checkpoints or parameter alignment.

3 Terminology and Definitions

The paper defines mode connectivity in weight space and introduces layer-wise variance spheres to describe the geometry used by its connection-building procedure. Independently trained modes are treated as low-loss parameter points from separate initializations.

  • Mode connectivity asks whether two independently trained modes can be joined by a continuous path that remains in low-loss regions.
  • A mode is a well-trained parameter point that attains low loss; the study focuses on modes from different random initializations.
  • The low-loss region consists of parameter points whose dataset loss does not exceed a specified threshold.
  • The full parameter point is decomposed into layer vectors, whose empirical means and variances characterize each layer.
  • Variance Sphere: A variance sphere is the set of layer parameters with fixed variance, and approximately fixes distance to the origin when layer means are near zero.
  • Variance Sphere: Independent initializations under the same architecture tend to produce trained modes on nearby variance spheres with layer-wise means near zero.

4 Architecture-aware Connection Building Algorithm

The architecture-aware connection building algorithm iteratively moves model layers toward a destination mode, corrects variance, and refines each intermediate point back toward the low-loss region. It adapts layer ordering to DDPM’s U-Net dependencies and NanoCLIP’s cross-modal structure, while using Adam to handle distant, high-dimensional checkpoints.

  • Two-stage connection building: Weighted Movement moves selected layers toward PE, while variance correction places the intermediate model on P0’s reference variance sphere.Averaging independently trained weights can reduce layer-wise variance and hinder subsequent training.
  • Two-stage connection building: The algorithm alternates weighted movement, variance correction, training refinement, and a second variance correction to generate consecutive low-loss anchor points.Repeating these stages for Ttotal iterations produces a sequence from P0 toward PT ≈ PE.
  • Dataflow-based layer moving strategy: The layer moving sequence Ui is the critical Weighted Movement hyperparameter because it determines which layer subset moves at each stage.Moving all layers simultaneously failed for the complex DDPM and NanoCLIP architectures.
  • Dataflow-based layer moving strategy: DDPM uses a symmetrical head-to-tail schedule that sequentially pairs corresponding encoder and decoder layers across its U-Net skip-connected paths.The strategy addresses the shared spatial resolutions of corresponding downsampling and upsampling blocks.
  • Dataflow-based layer moving strategy: NanoCLIP uses an alternating schedule that adds two visual encoder blocks followed by one text-encoder layer to preserve cross-modal coordination.The visual and textual encoders project features into a shared embedding space through contrastive learning.
  • Training refinement: Adam replaces unreliable global-learning-rate SGD refinement for large checkpoint separations by adapting updates to module-specific gradient magnitudes.Adam’s moment estimates provide coordinate-wise scaling and accumulate a stable refinement direction across updates.

5 Experimental Evaluation

Experiments evaluate architecture-aware paths between independently trained DDPM and NanoCLIP modes, testing path quality, endpoint behavior, baseline performance, and key design choices. The method maintains low loss across both models and outperforms AutoNEB and ablated alternatives.

  • Experimental Setup: Two independently trained modes are connected for DDPM on Flowers102 and NanoCLIP on Flickr30K using the proposed architecture-aware algorithm.The evaluation applies the method after independent training with different random seeds.
  • Overall Performance: Layer-wise distances decrease stage by stage for both models, with several final distances reduced by at least one order of magnitude from their initial values.The reductions reflect movement toward the end mode despite differing layer parameter scales.
  • Overall Performance: DDPM training loss stays below 0.04 and test loss converges near 0.04, while NanoCLIP training loss stays below 0.09 with stable training accuracy and Recall@1.These measurements indicate low-loss paths with preserved generation capability.
  • Overall Performance: Dense interpolation between consecutive NanoCLIP checkpoints reveals no observable barrier, supporting continuity of the discrete checkpoint path within the low-loss region.The check addresses hidden barriers that could occur between evaluated checkpoints.
  • Comparison Study: AutoNEB reaches DDPM maximum training and test losses of 0.17 and 0.18, versus 0.03 and 0.15 for the proposed method.On NanoCLIP, AutoNEB reaches maximum training loss 6.69 and minimum accuracy 0.05, while the method reaches 0.09 and 0.95.
  • Ablation Study: Moving all layers simultaneously raises average training loss to 0.031 on DDPM versus 0.025 for the complete method, and to more than twice the complete-method loss on NanoCLIP.The comparison indicates greater optimization difficulty when all layers move together.
  • Ablation Study: The proposed dataflow order reduces average training loss by 0.002 versus forward-order movement on both DDPM and NanoCLIP.Adam refinement also improves stability: DDPM loss is 0.024 versus 0.030 with SGD, while NanoCLIP loss can increase by two orders of magnitude with SGD.

6 Discussion

Discussion experiments show that low training loss along a mode-connected path does not guarantee high generation quality or locally flat endpoint geometry. Intermediate DDPM checkpoints can have worse FID, and nearby endpoint parameters can still produce a higher-loss interpolation.

  • Generation Quality: Across 41 DDPM checkpoints, intermediate models generally have higher FID than endpoints despite training losses remaining essentially endpoint-level.Each checkpoint generates 1000 images using identical sampling configuration and random seed.
  • Generation Quality: The results suggest that a low-loss region is not necessarily a high-quality generative region because refinement targets the loss constraint rather than FID.This separates the mode-connectivity criterion from direct generation-quality optimization.
  • Endpoint Geometry: A DDPM checkpoint with layer-wise cosine similarities near one to the endpoint still yields interpolation loss 0.038, versus 0.034 along the preceding low-loss path.The result is consistent with a potentially anisotropic low-loss region near the endpoint.

7 Conclusion

The paper extends mode connectivity to generative and contrastive models using an architecture-aware connection-building algorithm. Experiments construct continuous low-loss paths for DDPM on Flowers102 and NanoCLIP on Flickr30k.

  • The method combines dataflow-based layer movement with Adam-based refinement to build architecture-aware connections.The pipeline progressively reduces layer-wise parameter distances while maintaining low training loss.
  • Experiments show mode connectivity for independently trained DDPM models on Flowers102 and NanoCLIP models on Flickr30k.The constructed paths are continuous and remain in low-loss regions.
  • The work is presented as the first extension of mode connectivity to generative and contrastive domains.It motivates further study of loss-landscape geometry in modern non-classifier models.

A Empirical Validation of Equation 4

Across independently trained NanoCLIP and DDPM models, layer-wise variances remain consistent while layer-wise means stay centered around zero.

  • Layer-wise parameter variances remain consistent across independent training runs for NanoCLIP and DDPM models.The figure covers NanoCLIP on Flickr30k and DDPM on Flowers102 with different random seeds.
  • Layer-wise parameter means remain centered around zero across the independently trained models.Batch normalization layers are excluded because their statistics are highly sensitive to training batches.

B Mode-Connection Continuity Check via Linear Interpolation

Dense interpolation checks for NanoCLIP reveal no observable loss barrier, with all sampled points remaining in the low-loss region.

  • No observable loss barrier appears when twenty uniformly spaced interpolation points are inserted between consecutive optimization ticks.The continuity check evaluates the resulting NanoCLIP loss trajectory under denser sampling.
  • The maximum training loss increases by less than 5.3% along the constructed path.The interpolated models remain within the same low-loss region as the optimization trajectory.

C Hyperparameters and Configurations of Architecture-aware Connection Building Algorithm

The appendix reports the hyperparameters, staged layer-moving configurations, optimization schedules, and runtimes for the proposed algorithm and the AutoNEB baseline.

  • Table 2 specifies staged optimization configurations for the architecture-aware connection-building algorithm.Only the parameter subsets listed in each layer-moving sequence are optimized at a given stage, with total construction runtime also reported.
  • Table 3 reports a shared 14-cycle AutoNEB schedule for DDPM and NanoCLIP.The final path contains 13 internal pivots, while endpoint pivots remain fixed; total runtime is also reported.

E Experimental Results of Endpoint Parameter Proximity and Linear Interpolation

Figure 9 examines parameter-space proximity and training loss near the DDPM end mode. The checkpoint at tick 16400 is reported as highly close to the end mode in parameter space.

  • The DDPM checkpoint at tick 16400 is highly close to the end mode in parameter space.Layer-wise cosine similarities are summarized across multiple layers within each DDPM module.
  • The left panel reports layer-wise cosine similarity between the tick-16400 checkpoint and the end mode.Each box represents the distribution of cosine similarity values for multiple layers within one DDPM module.
  • The right panel presents training loss near the DDPM end mode, including loss along the original low-loss path.
Loading 2608.30366v1…