PaperlayerView exact revision on arXiv

Source-linked AI summary

Relative representations enable zero-shot latent space communication

Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, Emanuele Rodolà

arXiv:2209.15430v2cs.LGcs.AI

TL;DR

Neural-network latent spaces can differ across stochastic trainings, limiting direct comparison and component reuse despite preserving angular structure. The paper represents samples by similarities to anchors, yielding invariance to latent isometries and rescaling without additional training. This enables zero-shot model stitching and latent-space communication across trainings, architectures, datasets, and modalities.

  • Problem

    Stochastic training factors can produce incompatible latent spaces, hindering comparison and reuse even when data and modeling choices are fixed.

  • Method

    The method represents each sample by its similarities to a fixed set of anchor embeddings, using cosine similarity to preserve angle-based information.

  • Results

    The framework supports zero-shot stitching across distinct training regimens, architectures, datasets, and modalities, with relative representations outperforming incompatible absolute representations in reported applications.

  • Takeaways & Limitations

    Relative representations enable latent-space communication, allowing pretrained neural components to be compared and reused without trainable stitching layers.

  • Takeaways & Limitations

    The method uses cosine similarity and leaves the effects of alternative similarities, geodesic distances, and anchor-set composition for future research.

Abstract

from arXiv · show

Neural networks embed the geometric structure of a data manifold lying in a high-dimensional space into latent representations. Ideally, the distribution of the data points in the latent space should depend only on the task, the data, the loss, and other architecture-specific constraints. However, factors such as the random weights initialization, training hyperparameters, or other sources of randomness in the training phase may induce incoherent latent spaces that hinder any form of reuse. Nevertheless, we empirically observe that, under the same data and modeling choices, the angles between the encodings within distinct latent spaces do not change. In this work, we propose the latent similarity between each sample and a fixed set of anchors as an alternative data representation, demonstrating that it can enforce the desired invariances without any additional training. We show how neural architectures can leverage these relative representations to guarantee, in practice, invariance to latent isometries and rescalings, effectively enabling latent space communication: from zero-shot model stitching to latent space comparison between diverse settings. We extensively validate the generalization capability of our approach on different datasets, spanning various modalities (images, text, graphs), tasks (e.g., classification, reconstruction) and architectures (e.g., CNNs, GCNs, transformers).

1 INTRODUCTION

Neural networks can learn latent spaces that vary across trainings despite fixed data and modeling choices, although their angular relationships remain intrinsically similar. Relative representations are proposed to preserve this shared structure for comparing and reusing models.

  • Motivation: Latent spaces can change across trainings because of initialization, data shuffling, hyperparameters, and other stochastic factors.Repeated autoencoder trainings on MNIST produced different latent-space distributions despite fixed data and modeling assumptions.
  • Motivation: The learned representations remain intrinsically similar because distances or angles between embedded samples are approximately preserved.The paper attributes this symmetry to implicit optimization biases and notes that multiple spatial arrangements can satisfy the same similarity constraints.
  • Motivation: These changes make latent spaces difficult to compare and prevent direct reuse of components trained on different embeddings.Models may perform equally well on their tasks while remaining incompatible at the representation level.
  • Contribution: Relative representations encode each sample through similarities to anchors, depending only on angles between embeddings by construction.The approach uses a local coordinate system defined by other data samples rather than treating each embedding as an independent point in R^d.
  • Contribution: The framework introduces invariance to stochastic training transformations and supports zero-shot stitching, cross-setting latent comparison, and unlabeled performance measurement.The authors validate these contributions across different training regimens, architectures, modalities, and data settings.

2 RELATED WORK

Prior work studies representation similarity, model stitching, and relational feature transformations, but strong theoretical justification and direct component reuse remain limited. This paper avoids trainable stitching layers by using relative representations for zero-shot model stitching.

  • Representation similarity: Representation-similarity studies report comparable representations across architectures, tasks, and domains, although the phenomenon remains debated and lacks strong theoretical justification.The related work surveys evidence from neural and biological models while explicitly noting ongoing disagreement.
  • Model stitching: Model-stitching research has used trainable layers to swap network parts and evaluate whether different networks learn similar representations.Other approaches attempt to produce compatible reusable components directly, without stitching layers.
  • Model stitching: This work sidesteps trainable stitching layers and proposes zero-shot stitching to reuse models as frozen black-box modules.Relative representations provide the compatibility needed to combine components without additional stitching training.
  • Relative information: Attention, prototypical networks, and optimal-transport transforms represent samples using relationships among features or prototype representations.These methods provide precedents for exploiting relative information in learned representations.

3 METHOD

The method models stochastic changes in latent spaces as angle-preserving transformations and represents samples by similarities to selected anchors. Cosine similarity provides invariance to rotations, reflections, and rescaling, enabling latent-space comparison and model reuse.

  • Absolute representations: Standard neural networks map each sample x(i) to an absolute representation ex(i) in R^d for downstream tasks such as classification or reconstruction.The embedding function Eθ is optimized under a task objective with possible weight regularization.
  • Achieving latent invariance: Training stochasticity is modeled as a transformation T that changes embeddings while preserving pairwise angles between latent representations.The factors include initialization, data shuffling, hyperparameters, and other optimization randomness.
  • Relative representations: Relative representation rx(i) is a vector of similarities between sample x(i) and the embedded anchors in A.Each coordinate corresponds to one anchor, so the representation dimension equals the number of selected anchors.
  • Choice of the anchors: Anchor selection affects expressivity; classification tasks should sample anchors from each class, while parallel anchors can transfer partial correspondences across domains.For cross-domain data, corresponding anchors are obtained through a mapping Γ between subsets of the domains.
  • Choice of the anchors: Out-of-domain anchors can build relative representations for domain adaptation when correspondence is unavailable and labeled data are scarce.The approach uses elements from an external set with a pretrained encoder.
  • Achieving latent invariance: Cosine similarity is invariant to rotations, reflections, and rescaling, so the resulting relative representation is invariant to shared angle-preserving transformations.The translation case relies on the assumption that normalization techniques center latent spaces.
  • Applications: The invariance permits comparison across trainings and reuse of models with different latent embeddings.The paper demonstrates these applications in subsequent experiments.
  • Limitations and future work: Alternative similarity functions could enforce invariance to non-isometric bounded-distortion deformations, but this possibility remains future work.The authors did not find such invariance necessary in their experiments and report only preliminary investigation.

4 LATENT SPACE COMMUNICATION

Relative representations make latent spaces from different models comparable and support communication across semantically equivalent embeddings. Experiments show that relative-space similarity tracks model performance and that using relative representations during training generally avoids significant performance loss.

  • 4 LATENT SPACE COMMUNICATION: Relative representations enable communication between models trained on different data, architectures, or training conditions.The section evaluates word-embedding alignment, performance prediction, and end-to-end training with relative representations.
  • 4.1 WORD EMBEDDINGS: Relative embeddings align FastText and Word2Vec neighborhoods, whereas absolute embeddings have zero exact neighborhood matches.Relative neighborhoods match exactly 34% in one direction and 39% in the other, while absolute embeddings score zero on the Jaccard comparison.
  • 4.2 LATENT DISTANCE AS A PERFORMANCE PROXY: Similarity between relative latent spaces is a strong proxy for model performance across approximately 2000 Cora node-classification models.The mean Pearson correlation is 0.955 after filtering models whose best validation accuracy is below 0.5; the correlation also holds over time.
  • 4.2 LATENT DISTANCE AS A PERFORMANCE PROXY: The relative-space similarity metric is differentiable and can provide latent-space supervision without labeled data.The passage identifies teacher-student training as one possible use of this supervision signal.
  • 4.3 TRAINING WITH ABSOLUTE VS. RELATIVE REPRESENTATIONS: Relative representations used during end-to-end training are generally not detrimental to classification performance.Comparisons cover several image and graph datasets, with Table 2 reporting weighted F1 scores over six seeds.

5 ZERO-SHOT MODEL STITCHING

Relative representations enable frozen encoders and task-specific decoders to be stitched without training, across stochastic trainings, architectures, datasets, and modalities. Experiments report compatible reconstructions and zero-shot predictions where absolute representations fail or approach random guessing.

  • 5 ZERO-SHOT MODEL STITCHING: Relative representations enable zero-shot stitching of frozen neural components by composing an encoder with a downstream-task relative decoder.The operation requires no training or fine-tuning.
  • 5.1 IMAGE RECONSTRUCTION: Across five-seed autoencoder and variational-autoencoder experiments, relative representations produced nearly indistinguishable stitched and non-stitched reconstructions.Absolute representations instead produced erroneous predictions because independently trained latent spaces were incompatible.
  • 5.1 IMAGE RECONSTRUCTION: Relative stitching achieved reconstruction MSE up to two orders of magnitude lower than absolute stitching across five seeds.The metric is computed between ground-truth images and reconstructions.
  • 5.2 TEXT CLASSIFICATION: Cross-lingual experiments learned classification on one language and predicted ratings in another, while relative representations remained robust to noisy or distribution-shifted anchors.The evaluation used Amazon Reviews and mean weighted F1 and MAE across five seeds.
  • 5.2 TEXT CLASSIFICATION: Absolute representations yielded performances comparable to random guessing across cross-lingual and cross-architecture stitching, whereas relative representations showed invariance to architectures and training-language shifts.The reported cross-architecture settings included BERT, ELECTRA, and RoBERTa on TREC, DBpedia, and Amazon Reviews.
  • 5.3 IMAGE CLASSIFICATION: Relative representations supported stitching modules with different encoding dimensionalities and generalized to CIFAR-100 unseen by the ImageNet1k-frozen encoder.The decoder receives a relative representation with guaranteed equal size.

6 CONCLUSION

The paper concludes that relative representations enable zero-shot latent space communication through a simple representation shift when latent semantic correspondence exists. It also identifies several directions for improving invariance, expressivity, and multi-layer reuse.

  • 6 CONCLUSION: Relative representations exploit latent semantic correspondence between data domains through a simple representation shift without sophisticated processing or heavy training.The paper presents this as enabling zero-shot latent space communication.
  • 6 CONCLUSION: Future work includes alternative similarity functions, geodesic distances, anchor-set design, and stitching across multiple layers.The paper notes that anchor number and update frequency directly affect training cost.

A.1 HIGH-DIMENSIONAL LATENT SPACES

High-dimensional autoencoders produce intrinsically similar latent spaces across independent trainings, even though independently fitted visualizations can appear different. A shared PCA reference exposes differences caused by coordinate transformations rather than latent semantics.

  • A.1 HIGH-DIMENSIONAL LATENT SPACES: High-dimensional autoencoders trained independently produce intrinsically similar latent spaces across random seeds.The comparison uses latent spaces learned on MNIST with different seeds.
  • A.1 HIGH-DIMENSIONAL LATENT SPACES: Independently fitted PCA transformations make each latent-space visualization appear aligned, whereas applying PCA fitted on one space reveals differing coordinates.The first-row PCA is fitted separately per latent space; the second-row PCA is shared.

A.2 ANCHORS ANALYSIS

Anchor-set size and selection affect the quality of relative representations, with performance generally improving as anchors increase and selection strategies showing task-specific differences.

  • Anchor number: Anchor cardinality and identity are crucial: one anchor or repeated points can collapse the relative representation.The optimal anchor set remains an open empirical question, and each anchor corresponds to one relative-representation dimension.
  • Anchor number: Performance improves monotonically with more anchors when absolute representations are frozen, whereas end-to-end graph training is less stable.The comparison covers Cora node classification and frozen-transformer CIFAR-100 image classification.
  • Anchor selection: Uniform, farthest-point, k-means, and top-k strategies provide alternative ways to select anchors.Uniform sampling is random over available samples; farthest-point sampling uses geometric coverage, while k-means selects words near cluster centroids.
  • Anchor selection: Farthest-point sampling achieves the best Jaccard and MRR scores, while k-means achieves the best Cosine scores.The results are comparable across strategies, and k-means may be negatively biased toward high-density regions.
  • Anchor selection: Uniform sampling remains the most straightforward strategy because it requires no additional selection computation while maintaining good performance.The authors expect strategies covering the absolute space better to be most effective.

A.3 INVARIANCE WITH GUARANTEED BOUNDS

A modified similarity function uses vector quantization to make relative representations more similar across absolute embedding spaces while targeting robustness to bounded distortion.

  • Invariance with guaranteed bounds: Vector-quantized similarity produces more similar relative representations, measured by a lower average pairwise cosine-distance score.Agglomerative clustering of absolute embeddings at varying thresholds t performs the vector quantization.
  • Invariance with guaranteed bounds: The approach leaves open the trade-off between guaranteed invariance to arbitrary bounded distortion and representation expressiveness.This trade-off is identified as a topic for future work.

A.4 DATASET INFORMATION

The appendix summarizes the datasets used in the experiments and describes a word-embedding comparison between FastText and Word2Vec.

  • Dataset information: Table 9 lists each dataset together with its number of classes as an indication of classification difficulty.The passage does not enumerate the datasets or class counts.
  • Dataset information: Absolute FastText and Word2Vec representations appear dissimilar, while their relative representations yield almost identical spaces.The score is the average pairwise distance between embedding types, with lower values indicating greater similarity.

A.5 IMPLEMENTATION DETAILS

The appendix documents datasets, models, metrics, visualization procedures, experimental sweeps, and additional stitching and representation-similarity results.

  • Implementation details: Implementation details cover the experimental settings, tools, hyperparameters, transformers, preprocessing, and multilingual anchor construction.The appendix references PyTorch Lightning, Weights and Biases, HuggingFace Transformers and Datasets, and NN-Template GrokAI.
  • Visualization and settings: Word-embedding visualizations use 300 anchors and subsample 800 words from neighborhoods around four random pivot words.The procedure selects the top 200 neighbors for each pivot.
  • Visualization and settings: The computer-vision visualization uses 500 anchors and samples 200 images from each of four selected classes.The anchor count balances ViT-small and ViT-base encoding dimensions of 384 and 768.
  • Evaluation metrics: Evaluation compares source and target representations for approximately 20k samples using Jaccard and MRR metrics based on cosine-similarity neighborhoods and ranks.Final metric scores are averaged over samples.
  • Representation similarity: Relative representations preserve the absolute latent metric up to a certain degree while facilitating communication between aligned spaces.The analysis compares absolute spaces A and B with relative spaces C and D, including FastText and Word2Vec.
  • Experimental settings: Experiments vary random seeds for image classification and layer counts for graph classification, aggregating results by dataset and encoding type with standard deviations.Image sweeps use 10 seeds; graph configurations use 32 and 64 layers with 10 repetitions each.
  • Additional results: Additional appendix results cover latent-similarity correlations, multilingual and image stitching, multilingual-transformer stitching, and dataset and model tables.The cited tables report stitching performance for Amazon Reviews and CIFAR-100 settings.
Loading 2209.15430v2…