Source-linked AI summary
Unsupervised Latent Space Alignment with Hyperspherical Geodesic Matching
Cameron Ryan, Vivek Sivaraman Narayanaswamy, Kowshik Thopalli, Shusen Liu
TL;DR
Similar data can produce latent spaces with compatible geometry even when the spaces are not directly aligned, motivating alignment without paired anchors. HGA optimizes an orthogonal transformation for geometric fit and reports alignment comparable to supervised methods with little or no supervision, while its optimization remains unstable and becomes harder for increasingly complex spaces.
Problem
Independently trained models can produce similarly structured latent spaces, but alignment typically depends on shared sample anchors, raising whether geometry alone can recover the correspondence.
Method
HGA fits an orthogonal transformation by maximizing a geometric fit between two latent spaces, using intrinsic geometry rather than paired data and allowing unsupervised or weakly supervised alignment.
Results
HGA matches supervised alignment results with little or no supervision across model stitching and multilingual word-embedding correspondence tasks.
Takeaways & Limitations
Intrinsic latent-space geometry can often be sufficient for alignment without anchor supervision.
Takeaways & Limitations
HGA optimization is unstable, and finding useful objective maxima becomes more difficult as latent spaces become increasingly complex.
Abstract
from arXiv · showhide
Independently trained neural networks tend to encode the same data with similar latent geometries. These latent geometries are not directly compatible, yet they can be nearly the same up to some class of transformations. While there exists many methods for alignment between different latent spaces, it is typically done using a set of shared sample correspondences, known as anchors. This leaves a fundamental question: are the geometric signatures of different latent spaces representing similar data sufficient to recover an alignment between them? To that end, we introduce HGA (Hyperspherical Gaussian Alignment), a method that directly optimizes a transformation between two latent spaces by maximizing a geometric measure of "fit" between them. Since it is driven by the geometry of the latent spaces rather than paired data, HGA can operate in both an unsupervised and weakly supervised regime. On tasks such as model stitching or multilingual word embedding correspondence recovery, HGA manages to match supervised results with minimal or no supervision.
1 Introduction
Neural networks can encode similar data in latent spaces with nearly matching geometry, but aligning those spaces usually relies on shared sample anchors. HGA instead uses intrinsic geometry to seek alignment with no or limited supervision.
- Neural networks condense high-dimensional data into lower-dimensional latent spaces whose geometry reflects the underlying data manifold.
- Independently trained models often learn similar representations up to transformations such as isometries, rescaling, affine maps, or orthogonal maps.
- Anchor samples provide known correspondences between latent spaces and have enabled effective supervised alignment.
- HGA fits an orthogonal transformation that maximizes similarity between a mapped latent space and a target space.
- HGA is proposed for unsupervised and weakly supervised alignment, with intrinsic geometry often sufficient to determine the correspondence.
- Across datasets and modalities, HGA is compared with Latent Functional Maps and supervised Orthogonal Procrustes fitting, improving alignment with no or substantially less supervision.
2 Related Work
Prior work aligns latent spaces through model stitching, anchor-based weak supervision, spectral methods, and relational geometry. HGA belongs to the unsupervised or weakly supervised line that directly optimizes geometric fit between spaces.
- Model stitching evaluates representation compatibility by connecting layers from different networks with a learned transformation and measuring downstream performance.
- Anchor-based methods translate between latent spaces using shared samples, including relative representations that are invariant to latent isometries.
- Latent Functional Maps optimize a transformation between spectral spaces while preserving the Heat Kernel Signature.
- HGA operates in the weakly supervised or unsupervised regime by optimizing a transformation that maximizes geometric fit between latent spaces.
- Optimal transport compares embedding distributions, while Gromov-Wasserstein distance avoids unknown cross-space costs by comparing intra-space pairwise distances.
3 Method
HGA normalizes latent samples to a unit sphere and searches over orthogonal transformations for the highest cross-space geometric affinity. Anchor relationships can be incorporated through weighted objective terms or initialization.
- HGA formulation: HGA assumes corresponding points in separate latent spaces have approximately equal relative angles and seeks alignment by rotating or reflecting one point set.
- HGA formulation: The method normalizes both datasets to the unit sphere and optimizes an orthonormal matrix R to fit the mapped and target spaces.
- HGA formulation: HGA converts cosine similarities into spherical geodesic distances and then into an unnormalized Gaussian kernel between points.
- Optimization: The objective averages kernel affinities across all point pairs, and gradient ascent searches for argmax_R∈O(d) Jσ(R) from a random orthogonal initialization.
- Supervision: Known anchors can receive higher objective weights, while an orthogonal Procrustes fit can initialize the optimization from the anchor correspondences.
- Supervision: The anchor-weight hyperparameter is often set near w ≈ m × n, but values that are too large can destabilize the objective landscape.
4 Experiments
Experiments evaluate HGA for model stitching and multilingual word-embedding translation. The study tests whether unsupervised geometry can approach anchor-supervised alignment using neural-network representations and FastText embeddings.
- Model Stitching: Model stitching inserts a learned transformation between layers of two networks and evaluates the resulting stitched model on the downstream task.
- Model Stitching: For CIFAR-10 and CIFAR-100 experiments, pairs of identical models are aligned using a subset of 1024 samples, and unsupervised HGA often achieves stitching accuracy similar to using all anchors.
- Word Embedding Translation: FastText experiments align English and French embedding spaces using manually created word correspondences as anchors and evaluate accuracy against MUSE dictionaries.
- Word Embedding Translation: Figure 1 compares word correspondence accuracy against the number of anchors for multiple alignment methods on FastText embeddings.
5 Discussion and Conclusion
HGA aligns independently trained latent spaces by maximizing a geodesic-kernel measure of geometric fit, matching supervised methods with little or no supervision. Its main limitations are unstable optimization, increasing difficulty on complex spaces, and substantial computational cost.
- HGA fits an orthogonal transformation that maximizes a geodesic-kernel measure of geometric fit between independently trained latent spaces.
- HGA can match supervised alignment methods with little or no supervision.
- Optimization is unstable because large initial bandwidths can create incorrect distribution matches and false alignments.
- For simple datasets such as MNIST or CIFAR10, HGA can achieve stitching accuracy similar to fully supervised approaches, whereas complex spaces often still require anchors.
- Finding useful maxima becomes harder as latent spaces become more complex, and current optimization takes about an hour to find a decent alignment.Future work targets parallelized restarts, improved early stopping, and geometric characterization of latent spaces.
A Optimization Method
The optimization searches over orthogonal transformations using projected gradient ascent, retraction onto the orthogonal group, and a decreasing bandwidth schedule. Multiple random restarts address undesirable local optima, while early stopping can reduce computation after promising runs emerge.
- Optimization starts from a random orthogonal matrix and iteratively improves it with gradient ascent.The Euclidean gradient is projected into the tangent space of O(d), and the updated matrix is mapped back using singular value decomposition.
- The Euclidean gradient is projected into the tangent space of O(d) before updating the transformation and retracting it onto the orthonormal group.
- The bandwidth σ is decreased through a sequence initialized from the previous solution, moving from smooth global alignment toward local point alignment.At large σ the objective is highly smooth and nearly concave; at smaller σ it considers nearby pairs and becomes highly nonconcave.
- A cosine annealing decay controls optimization hyperparameters between ηmin and ηmax.
- Many random restarts are needed because changing bandwidth can create undesirable local maxima from symmetries that disappear during optimization.
- Early stopping can terminate low-probability runs near the final optimization stage when their objective scores fall below an experiment-specific threshold.
B Hyperparameter Experiment
The hyperparameter experiment evaluates how configuration changes affect HGA model-stitching accuracy across every layer of a CIFAR10-trained model. The study records the configurations producing the best observed accuracies.
- HGA hyperparameter variations can have inexplicably large effects on model-stitching accuracy.
- The experiment tests several variations of each hyperparameter for model stitching at every layer of a CIFAR10-trained model.
- The study records the hyperparameter configurations associated with the best stitching accuracy after comparing the tested variations.
B.1 Anchor Corruption Test
The anchor-corruption experiment tests whether HGA remains effective when correspondences are progressively replaced with points from incorrect classes. On MNIST latent spaces, stitching accuracy remains nearly unchanged with up to 85% corrupted anchors.
- HGA maintains nearly unchanged MNIST stitching accuracy even when up to 85% of its anchors are corrupted.
- The experiment progressively replaces correct anchors with correspondences to points from incorrect classes and compares alignment methods by stitching accuracy.
- Figure 2 plots objective-function score at σ = 0.2 against optimization step for 10 random orthogonal initializations, with color indicating final score.Visible jumps occur when σ is decreased for the objective used to compute the optimization gradient.
C Verifying the Usefulness of the Objective Function for Unsupervised Alignment
The authors test whether HGA’s objective value identifies useful alignments by comparing optimization scores with model-stitching accuracy and examining kernel-based correspondences.
- Objective-function validation: The experiments assess whether higher objective-function values correspond to desirable model-stitching alignments.Optimization trials use varied hyperparameters on CIFAR10 and CIFAR100, with stitching accuracy evaluated for each resulting transformation.
- Objective-function validation: For σ ≈0.2, objective-function value strongly correlates with stitching accuracy when objective values are high.
- Objective-function validation: For larger σ, a high objective-function value is necessary but not sufficient for high stitching accuracy.
- Unsupervised stitching: The kernel recovers correspondences between points from their latent-space positions across MNIST, CIFAR10, and CIFAR100.Figure 5 presents kernel subsets and model-stitching confusion matrices for unsupervised stitching.
E Qualitative Results for FastText Alignment
The FastText experiment evaluates whether HGA can use limited word anchors to infer additional correspondences between embedding spaces.
- Qualitative FastText results: With 100 anchor words, HGA infers correspondences between words absent from the anchor list.Both HGA and LFM represent known word correspondences, while HGA also extends alignment beyond the provided anchors.
F Experiments with Distribution Matching
The experiments use kernel-based distribution matching to identify mismatches between latent spaces and compare image encoders with different training data. These analyses show that kernel sums can reveal missing classes and concepts associated with textual information.
- Distribution matching: Kernel-based distribution matching is used to investigate whether separate latent spaces match.The setup compares two spaces through a transformation and kernel function.
- Missing class detection: Samples from the MNIST model’s missing “0” class have very low kernel values against the regular model’s latent space.The excluded class also has much lower kernel-sum values than regular classes.
- Missing class detection: Kernel sums locate a target-space region that is present in one latent space but absent from the other.This provides a geometric signal for detecting representations of a class missing during training.
- Image embedding model alignment: In image-encoder comparisons, concepts with the lowest kernel sums tend to be most related to textual information.The comparison uses DINO-V2 as the image-only source and CLIP-image as the image-and-text target.