PaperlayerView exact revision on arXiv

Source-linked AI summary

Harnessing the Universal Geometry of Embeddings

Rishi Jha, Collin Zhang, Vitaly Shmatikov, John X. Morris

arXiv:2505.12540v4cs.LG

TL;DR

Different text embedding models produce incompatible spaces, limiting translation when paired data and the original encoder are unavailable. The paper introduces vec2vec, which translates embeddings through a learned universal latent representation using unsupervised objectives. It achieves cosine similarity as high as 0.96 and enables sensitive information extraction from unknown embeddings, while current results remain preliminary and dependent on future methodological improvements.

  • Problem

    Different models encode text into incompatible vector spaces, while unsupervised translation lacks paired data and access to the original encoder.

  • Method

    vec2vec uses space-specific adapters, a shared backbone, adversarial losses, and cycle consistency to translate embeddings through a universal latent representation.

  • Results

    0.96 cosine similarity to ground-truth target vectors and perfect matching on over 8000 shuffled embeddings were achieved without predefined matches.

  • Takeaways & Limitations

    Translated embeddings preserve sufficient semantics for zero-shot attribute inference, including sensitive disease information and partial corporate-email content.

  • Takeaways & Limitations

    The reported results are preliminary and future improvements are needed to scale translation to more data, model families, and modalities.

Abstract

from arXiv · show

We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches. Our unsupervised approach translates any embedding to and from a universal latent representation (i.e., a universal semantic structure conjectured by the Platonic Representation Hypothesis). Our translations achieve high cosine similarity across model pairs with different architectures, parameter counts, and training datasets. The ability to translate unknown embeddings into a different space while preserving their geometry has serious implications for the security of vector databases. An adversary with access only to embedding vectors can extract sensitive information about the underlying documents, sufficient for classification and attribute inference.

1 Introduction

Text embeddings encode semantics but remain incompatible across models. vec2vec learns unsupervised translations through a universal latent structure, achieving strong similarity while exposing semantic information in unknown embeddings.

  • Motivation: Different embedding models encode semantically related texts in incompatible vector spaces despite embeddings powering core NLP tasks.These embeddings support retrieval, RAG, classification, and clustering.
  • Universal representation: vec2vec proposes learning and harnessing a universal latent text representation to translate between model spaces without paired data or encoders.The proposal extends the Platonic Representation Hypothesis constructively to text models.
  • Evidence: Given unpaired embeddings from models with different architectures and training data, the method learns a latent representation in which embeddings are almost identical.This result is presented as practical evidence for the Strong Platonic Representation Hypothesis.
  • Method: Adversarial losses and cycle consistency encode embeddings into a shared latent space and decode them with minimal loss.Vector space preservation is also used to map an unknown embedding distribution to a known one.
  • Results: 0.96 cosine similarity to ground-truth target vectors and perfect matching on over 8000 shuffled embeddings demonstrate accurate translation without predefined matches.The reported matching result uses no access to the possible match set in advance.
  • Security implications: Translated embeddings preserve enough semantics for zero-shot attribute inference and inversion without knowing the original embedding model.These analyses test semantic preservation rather than geometry alone.

2 Problem formulation: unsupervised embedding translation

The task is to extract information from embeddings produced by an unknown encoder when paired representations and encoder access are unavailable. vec2vec instead relies on an accessible reference encoder and shared geometry between embedding spaces.

  • Problem setup: The problem starts with a vector database whose embeddings come from an unknown encoder and whose underlying documents are unavailable.The encoder cannot be queried, and its training data and architecture are unknown.
  • Available information: The attacker may query a different encoder and knows only high-level properties such as the documents’ modality and language.Translated embeddings can then be processed by tools requiring the known encoder.
  • Why matching is insufficient: Correspondence methods usually require candidate embeddings generated from the same or overlapping inputs, which is unrealistic for many databases.Some methods support non-overlapping inputs, but experiments report that they struggle significantly even when correspondence exists.
  • Scope: Prior word-embedding translation commonly relies on overlapping vocabularies, whereas this task translates embeddings of entire sequences.The distinction broadens translation beyond word-level correspondences.
  • Core challenge: Unsupervised translation is harder than matching because neither the original encoder nor additional document representations are available.Its effectiveness therefore depends on shared geometric structure across the two embedding spaces.
  • Hypothesis: The Strong Platonic Representation Hypothesis conjectures that models with the same modality and objective converge to a universal latent space enabling translation without pairwise correspondence.Training data and model architectures may differ under this conjecture.
  • Consequence: Successful translation would allow vector-based information-extraction tools, such as inversion models, to operate on embeddings from an unknown encoder.The proposed application is recovery or extraction of information about the hidden documents.

3 Our method: vec2vec

vec2vec uses space-specific adapters and a shared backbone to translate embeddings through a common latent representation. Training combines adversarial distribution matching with reconstruction, cycle consistency, and vector-space preservation.

  • Architecture: The architecture maps each encoder-specific embedding space into a shared latent space through input adapters, a backbone, and output adapters.The shared backbone extracts a common latent embedding before decoding into either model-specific space.
  • Architecture: Translation and reconstruction functions compose input adapters, the shared backbone, and output adapters between the two embedding spaces.The mappings include cross-space translations and same-space reconstructions.
  • Network design: Because embeddings lack spatial bias, vec2vec uses multilayer perceptrons with residual connections, layer normalization, and SiLU nonlinearities.Discriminators omit residual connections to simplify adversarial learning.
  • Adversarial objective: Adversarial training matches generated embeddings to the empirical distributions of original embeddings at both embedding and latent levels.Discriminators operate on latent representations and output embeddings.
  • Generator constraints: Adversarial losses alone do not guarantee semantic preservation, so the generator receives reconstruction, cycle-consistency, and vector-space-preservation constraints.These constraints help the mapping remain useful while translating between spaces.
  • Reconstruction: Reconstruction requires mapping an embedding into latent space and back to reproduce its initial representation.The reconstruction loss compares the recovered representation with the original embedding; p and q denote distributions sampled from the two encoders.
  • Cycle consistency: Cycle consistency makes translation to the other space and back return the original embedding with minimal corruption.It serves as an unsupervised proxy for paired alignment.
  • Geometry preservation: Vector space preservation keeps pairwise relationships among translated embeddings consistent with those in the target space.The loss aggregates average pairwise distances across batches after translation.

4 Experimental setup

The experiments evaluate unsupervised translation across diverse embedding models, using NQ for training and evaluation and separate datasets for information extraction. Translation quality is measured against target embeddings and baselines, while semantic preservation is assessed through attribute inference and embedding inversion.

  • Datasets: The study uses NQ queries and Wikipedia answers, with a 2-million training subset and a 65,536-example evaluation subset.
  • Models: The model suite spans four size categories, five transformer backbones, and two output dimensionalities, including multilingual Granite and multimodal CLIP.
  • Training: Each vec2vec model is trained on disjoint sets of 1 million 64-token NQ sequences, with the best of multiple initializations selected because of GAN instability.
  • Translation evaluation: Translation quality is evaluated by matching translated source embeddings against unknown target embeddings and reporting mean cosine similarity, top-1 accuracy, and mean rank.
  • Translation evaluation: The comparison includes a Naïve identity baseline and an Oracle-aided optimal-transport baseline that assumes candidate targets are known.
  • Information-extraction evaluation: Information extraction is evaluated through zero-shot attribute inference and embedding inversion, with an Oracle attribute-inference comparison using the correct target embeddings and labels.

5 vec2vec learns to translate embeddings without any paired data

vec2vec learns a universal latent space that preserves embedding geometry across model pairs, including out-of-distribution data and multimodal CLIP embeddings. These translations outperform baselines, especially for cross-backbone pairings.

  • 5 vec2vec learns to translate embeddings without any paired data: vec2vec maps embeddings from different models into nearly identical representations in a shared latent space.The method projects embeddings through input adapters and a shared translator; embeddings that are far apart can have nearly overlapping latents.
  • 5 vec2vec learns to translate embeddings without any paired data: vec2vec translations achieve near-optimal assignment across model pairs, with cosine similarity up to 0.92 and top-1 accuracy up to 100%.Cross-backbone pairings show the largest advantage over baseline methods, which perform similarly to random guessing.
  • 5 vec2vec learns to translate embeddings without any paired data: vec2vec retains high cosine similarity, accuracy, and low rank when trained on NQ and evaluated on tweets and medical records.These datasets differ from NQ in style and domain, including colloquial language, emojis, and medical jargon.
  • 5 vec2vec learns to translate embeddings without any paired data: The translations preserve enough target-space geometry to support zero-shot attribute inference.Table 5 evaluates information leakage using TweetTopic and MIMIC records, with same-model embeddings serving as an ideal baseline.
  • 5 vec2vec learns to translate embeddings without any paired data: For CLIP, vec2vec consistently outperforms the optimal transport baseline, although translations are weaker than those in the text-only experiments.The result connects unimodal text embeddings with a multimodal embedding space despite differing embedding dimensionalities.

6 Using vec2vec translations to extract information

vec2vec translations preserve semantic information as well as geometric structure. This enables attribute inference and text inversion from translated embeddings, including extraction of potentially sensitive details.

  • 6 Using vec2vec translations to extract information: vec2vec translations consistently outperform the naïve baseline for zero-shot attribute inference and often exceed the ideal zero-shot baseline.The ideal baseline performs inference using ground-truth document and attribute embeddings in the same space.
  • 6 Using vec2vec translations to extract information: Attribute inference remains effective on medical records, with cross-backbone translations often greatly outperforming the naïve baseline.The evaluated attributes are MedCAT disease descriptions, few of which occur in the training data.
  • 6 Using vec2vec translations to extract information: Figure 5 measures information leakage through inversion on a 50-email Enron subset using a model trained on NQ.Cells report judge accuracy.
  • 6 Using vec2vec translations to extract information: Figure 6 shows inversion examples that infer entities and content from Enron emails.The examples illustrate the kinds of document information recoverable from translated embeddings.
  • 6 Using vec2vec translations to extract information: Zero-shot inversion extracts information from as many as 80% of emails and 67% of tweets using only translated embeddings.The inversions can reveal names, dates, promotions, financial information, outages, and lunch orders, though they are imperfect.

7 Ablations

Ablations indicate that vec2vec’s components are important for good translations, while substantially reducing training data still produces non-random results and near-full-data performance at 50K embeddings.

  • 7 Ablations: Removing individual vec2vec components degrades translation quality and geometry preservation across the evaluated metrics.The full method’s cosine similarity exceeds the naïve baseline, but removing components performs worse than the optimal transport baseline.
  • 7 Ablations: With as few as 10K embeddings, translators learn more than random, while 50K embeddings produce results almost as good as 1M.Translations generally improve as the amount of training data increases.

8 Related work

Prior work studies representation alignment, unsupervised transport, embedding inversion, and modality bridging, but vec2vec targets translation between unknown embedding spaces without paired inputs.

  • 8 Related work: Representation-alignment methods support comparison, stitching, substitution, domain transfer, and multimodal adaptation, but rely on some paired data.The cited approaches include CCA, SVCCA, CKA, ICA, time-series methods, and GUIs.
  • 8 Related work: Unsupervised optimal transport methods inspire vec2vec through cycle consistency and adversarial loss, while prior word and sequence translation settings had overlapping representations unavailable here.vec2vec applies these ideas to entire-sequence embeddings without paired inputs.
  • 8 Related work: Embedding-inversion research decodes text from embeddings, whereas vec2vec enables such methods on unknown embeddings without an encoder or paired data.The translation step brings unknown embeddings into the space of a known model.
  • 8 Related work: Modality-bridging work unifies image, text, audio, and vision representations, while vec2vec post-processes embeddings to connect spaces.The paper evaluates this idea by translating between text and CLIP embeddings.

9 Discussion and Future Work

The paper argues that a universal latent representation enables unsupervised translation across embedding spaces, while showing applications to multimodal retrieval and information extraction. It also identifies computational and training-stability limitations and leaves broader scaling for future work.

  • The Strong Platonic Representation Hypothesis proposes that representation spaces can be translated through a universal latent structure without paired data or encoders.
  • vec2vec preserves sufficient input semantics to enable attribute inference from translated embeddings alone.The method extracted sensitive disease information from patient records and partial content from corporate emails without access to the original encoder.
  • The paper leaves scaling to more data, model families, and modalities to better and more stable learning algorithms, architectures, and other methodological improvements.The authors describe the reported results as a lower bound on inter-representation translation.
  • 176 GPU days of training and 42 GPU hours of evaluation and analysis were consumed, with an additional 45 CPU hours for optimal transport.
  • Translating Granite embeddings to CLIP enabled non-negligible cross-model multimodal retrieval despite zero multimodal training.The experiment used a unimodal Granite-to-CLIP translator on MS COCO captions and images.
  • Further evaluation with multimodal-specific training is identified as a promising direction.

E Initialization robustness by model backbone

vec2vec training is relatively stable for related model backbones but substantially less stable for unrelated ones under a fixed 10-epoch budget.

  • 14 of 15 related-model runs achieved at least 80% top-1 accuracy within the fixed epoch budget, while the remaining run reached 72%.
  • Only 3 of 15 unrelated-model runs achieved convergence at 80% top-1 accuracy.
  • The authors leave improving seed stability of the training regime as future work.

F Full out-of-distribution translation results

The full out-of-distribution evaluation tests models trained on NQ against tweets and medical records, and reports non-negligible information leakage from translated tweet embeddings.

  • vec2vec models trained on NQ were evaluated on the entire TweetTopic test set of 800 tweets and on an 8192-record MIMIC subset.
  • The rank metric ranges from 1 to 800 for TweetTopic, with 400 representing a random ordering.
  • The rank metric ranges from 1 to 8192 for MIMIC, with 4096 representing a random ordering.
  • vec2vec achieved non-negligible leakage for all model pairs when extracting information from translated tweet embeddings.
  • Figures 5 and 7 used an LLM judge to assess whether approximate inversion of translated email embeddings leaked information about the original email.
Loading 2505.12540v4…