Source-linked AI summary
Computation Over Geometry: Meaning Identity Is Computed, Not Shipped in the Embeddings
Jiaqi Deng
TL;DR
The paper asks whether paraphrase identity resides in independently encoded sentence vectors, as retrieval and RAG often assume. It compares independent-vector readouts with joint-pass computation on overlap-matched PAWS-X and finds identity is readable in the joint pass, not the shipped embedding geometry. The paper also reports that this relation can be distilled into a smaller joint reader.
Problem
Retrieval and RAG often treat meaning identity as a geometric property of independent sentence vectors, but evidence is limited on whether paraphrase identity is readable from those vectors under overlap-matched evaluation.
Source: §1 · pp. 1–2Method
Results
0.90–0.96 joint confirm AUC versus 0.47–0.61 for separate cosine, separate pair probe, and late fusion on overlap-matched English PAWS-X; partner shuffle returns to ≈0.50.
Source: §1 · p. 2Takeaways & Limitations
Meaning identity is a computed relation rather than a property of frozen independent embeddings, and a 1.5B joint reader can distil it from teacher scores.
Source: §1 · p. 3, §11 · p. 10The evidence is scoped to PAWS-style paraphrase under overlap matching, where independent-encoding readers trail joint probes even with 49k training pairs.
Source: §10 · p. 10
Abstract
from arXiv · showhide
Meaning identity (whether two sentences say the same thing after wording changes) is treated in retrieval and RAG as a geometric fact about independently encoded sentence vectors. We show that, for frozen off-the-shelf encoders and language models, it is not: identity is computed when both sentences share one forward pass, and is not a property of the embedding geometry those systems ship. On overlap-matched PAWS-X, purpose-built encoders (BGE, E5, GTE, MiniLM, E5-Mistral-7B) reach English confirm AUC only 0.55-0.65 (dense peak 0.70). Independently encoded last-token states of Llama 3, Mistral, and Qwen do no better; late fusion of the two vectors stays near chance. The same probe on a joint forward pass reaches 0.90-0.96 from 1.5B to 32B, collapses under partner shuffle, is mid-depth, saturates near 0.94 by 3B, and appears more weakly in GPT-2 XL (0.76). The gap holds beyond Llama-style models on other causal LMs, bidirectional encoders (DeBERTa, RoBERTa), and encoder-decoders (Flan-T5, T5, BART). Fixed or linear readers over frozen independent encodings never unlock identity; nonlinear pair readers recover part of it only on the full 49k-pair PAWS train split (0.68-0.87). Off-the-shelf rerankers split: BGE-reranker-large reaches 0.94, while MS-MARCO and Jina stay at 0.55-0.64. Independently trained families compute the same relation and a 1.5B joint reader can distill it from unlabelled teacher scores, while no linear function of the teachers own independent vectors can. Bi-encoders can be fine-tuned to fit PAWS (0.87-0.93), but transfer and STS-B suffer. Cosine compares wording neighbourhoods; identity is a cheap computed operator, not a property of either sentence vector.
1 Introduction
The paper challenges the assumption that meaning identity can be read from independently encoded sentence vectors, even when embeddings remain useful for retrieval and semantic similarity. It contrasts that geometric view with identity computed when sentence pairs share a forward pass.
1 Introduction
- Sentence embeddings support retrieval, clustering, and STS, but may miss PAWS’s distinction between high-overlap paraphrases and non-paraphrases (Cer et al., 2017; Zhang et al., 2019; Yang et al., 2019).The assumption that identity belongs to a single sentence representation underlies cosine-based comparison in retrieval and RAG (Reimers and Gurevych, 2019; Muennighoff et al., 2023; Gao et al., 2024).Source: §1 · p. 1
- Identity is readable from the last token of a joint pass, but not from a fixed or linear function of the two separately encoded states.The shuffled-partner control preserves format and sentence-B marginal while destroying the pairing.Source: §1 · p. 2
- 0.90–0.96 joint confirm AUC versus 0.47–0.61 for separate cosine, separate pair probe, and late fusion on overlap-matched English PAWS-X; partner shuffle returns to ≈0.50.The joint signal appears at mid-depth across modern open families; dedicated encoders remain at 0.55–0.70, and GPT-2 XL reaches 0.76.Source: §1 · p. 2, Figure 1 · p. 3
- A 1.5B joint reader distilled from three cross-family teachers matches the 7B teacher on PAWS (0.959) and raises trapped FAQ top-1 from 0.47 to 0.88 without changing the retrieval index.The reader is trained on teacher consensus over unlabelled text.Source: §1 · p. 3
- The paper argues that embeddings should not be asked to certify identity: doing so asks a bi-encoder to do a cross-encoder’s job.The authors frame identity as a computed operator rather than a property of either shipped sentence vector.Source: §1 · p. 3
2 Related work
Prior work distinguishes bi-encoder retrieval from pairwise reranking, but the paper asks whether paraphrase identity is accessible from independent sentence representations at all. Its overlap-matched evaluation is designed to test that question beyond word-overlap cues.
2 Related work
- Cross-encoders are known to outperform bi-encoders on reranking, but the paper argues the gap is sharper than a compute/quality tradeoff: only some joint computations implement PAWS-style identity.BGE rerankers reach the joint-probe band, whereas MS-MARCO and Jina rerankers do not.Source: §2 · pp. 3–4
- Fixed or linear readers over two frozen encodings stay near chance at any training size, while pair computations trained from scratch need forty times the pairs of a joint-pass probe to approach it.The claim concerns accessibility to trained readouts, not information-theoretic absence.Source: §2 · p. 4
- The paper’s question precedes work mapping linearly readable lexical, semantic, and syntactic signals: is paraphrase identity a function of two independent points at all?Related work includes probing and representation hypotheses (Alain and Bengio, 2017; Park et al., 2024), as well as depth and overlap studies (Li and Subramani, 2026; Acevedo et al., 2026; Rizwan et al., 2026).Source: §2 · p. 4
- The evaluation matches lexical overlap between paraphrase and non-paraphrase PAWS-style pairs, so an overlap-only scorer is at chance.PAWS and PAWS-X adversarialise bag-of-words overlap (Zhang et al., 2019; Yang et al., 2019); the paper retains this setting to test identity under overlap stress.Source: §2 · p. 4
3 Protocol
The protocol compares four readouts on identical frozen-model pairs, using overlap-matched PAWS-X splits and confirm-only reporting. Controls and fixed evaluation choices distinguish pair-specific computation from geometry, templates, and overlap cues.
3 Protocol
- PAWS-X English and Chinese pairs are split into discovery, development, and confirm sets, with lexical overlap matched within each language-by-split cell.English confirm overlap gaps are typically below 0.02; AUC is used so constant anisotropic offsets cannot change the result, and confirm intervals use bootstrap 95% confidence intervals.Source: §3 · p. 4
- Four readouts score the same pairs: independent-state cosine, a separate pair probe, late fusion of both vectors, and a probe on the last token of a concatenated joint pass.Probes are fit on discovery, regularisation is selected on development, and scores are reported on confirm; peak layers are selected by confirm joint AUC.Source: §3 · p. 4
- The shuffle control replaces sentence B with another sentence from the same language and split, preserving format and its marginal while destroying the original pairing.A probe responding only to the presence of two sentences should survive the shuffle.Source: §3 · p. 4
- Depth measurements cover a fixed grid of relative depths and both final-norm input and output states.The protocol separately scores zero-shot verdict, length-normalised PMI, and conditional surprisal on a smaller overlap-matched draw.Source: §3 · p. 5
- Nineteen dense checkpoints and six cross-encoders are evaluated on the same dedicated overlap-matched bank, with 900 confirm pairs per language.The preregistered readings predict joint-over-late-fusion separation and a shuffle drop to chance; the first depth whose confirm 95% interval excludes 0.55 is reported.Source: §3 · p. 5
4 Geometry does not carry identity
On overlap-matched PAWS, frozen independent embeddings and several independent-encoding readers fall short of joint-pass probes, while some rerankers and task-fine-tuned encoders do reach high scores. The results distinguish pairwise identity computation from a general-purpose embedding geometry.
4 Geometry does not carry identity
- 0.701 is the strongest English cosine among dedicated encoders; typical retrieval encoders reach 0.55–0.65, and E5-Mistral-7B reaches 0.613.Linear probes barely improve BGE-small (0.588 to 0.607) and BGE-large-zh (0.701 to 0.710).Source: §4 · p. 5
- 0.940 for BGE-reranker-large, 0.902 for v2-m3, and 0.888 for base, versus 0.639 for Jina reranker-v2 and 0.545–0.558 for MS-MARCO MiniLM rerankers.Joint scoring alone is insufficient: the latter rerankers remain in the dense band and score passage relevance rather than PAWS-style identity.Source: §4 · p. 5
- 0.52–0.61 for separate cosine and 0.51–0.62 for separate pair probes across the tested language models; late fusion stays at 0.47–0.57 for modern 3–14B models.Qwen2.5-14B late fusion reaches 0.541, while its cosine of 0.614 remains far below its joint probe.Source: §4 · p. 5
- 0.62–0.71 for last-layer token MaxSim, but a two-layer MLP on concatenated last tokens trained on 1.2k discovery pairs stays at 0.47–0.56.MaxSim saturates early: Qwen2.5-32B reaches 0.694 versus joint 0.938, and Phi-2 reaches 0.701 versus joint 0.893.Source: §4 · p. 5
- A cross-attention reader over frozen token sequences reaches 0.68–0.70 on several models and 0.87 on Qwen14B only with the full 49k-pair train split.The same reader fits a relatedness control at 0.98–0.99; an MLP on pooled pairs reaches 0.72–0.75, while the joint probe reaches 0.90–0.95 from 1.2k pairs.Source: §4 · pp. 5–6
- Figure 2 shows the joint readout rising at mid-depth while the other three readouts stay near chance; in postnorm, Qwen14B shifts from 0.950 to 0.940 and Llama from 0.897 to 0.731.Source: Figure 2 · p. 6
- MiniLM, BGE-small, and BGE-base fine-tuned on PAWS-X reach 0.871, 0.884, and 0.930 on overlap-matched confirm, but their QQP and STS-B performance falls.QQP scores fall from 0.813 to 0.701, 0.824 to 0.739, and 0.829 to 0.693; STS-B Spearman falls from 0.87–0.90 to 0.61–0.62.Source: §4 · p. 6
5 Identity appears when the pair is computed
Across models, identity is strongly readable from a joint pass but not from separately encoded vectors; shuffle and cross-task tests show the relation depends on the paired sentences.
5 Identity appears when the pair is computed
- 0.37–0.46 AUC: for Llama, Mistral, and Qwen ≥1.5B, joint probes exceed late fusion, while partner shuffle returns to 0.46–0.54.Table 1 reports the readouts at each model’s peak joint layer.Source: §5 · p. 6
- 0.921–0.953: Flan-T5-base, nli-DeBERTa, and DeBERTa-v3 also show joint-over-late-fusion peaks, with partner shuffle near chance.Smaller T5/BART models show the same-signed pattern at 0.64–0.66.Source: §5 · p. 7
- 0.919–0.967: Qwen2.5 joint AUC spans this range across four concatenation templates at 7B–32B, and every wrapper beats late fusion and collapses under shuffle.Mistral and Llama at off-peak layers are more wrapper-sensitive, but retain that pattern.Source: §5 · p. 7
- 0.80–0.83 joint AUC on anchor-balanced QQP, versus 0.55–0.60 late fusion, 0.58–0.62 cosine, and 0.42–0.46 shuffled joint.Each anchor appears once with a duplicate and once with an overlap-matched non-duplicate, removing label information from the first sentence.Source: §5 · p. 7
- 0.72–0.75 on QQP and 0.745–0.818 on Chinese PAWS: a PAWS-trained joint direction transfers zero-shot, though less strongly than its 0.88–0.91 in-domain QQP scores.The reverse QQP-to-PAWS transfer is also reported, while transfer is partial and some probe signal is dataset-specific.Source: §5 · p. 7
- 0.85–0.90 PAWS confirm: a 1.5B joint student reaches this band from teacher scores without gold labels, while ridge fits over independent states score 0.46–0.62.Adding teacher-scored unlabelled QQP pairs improves zero-shot QQP scores, including 0.71→0.79 under Mistral and 0.69→0.73 under Llama.Source: §5 · pp. 7–8
- Mid-depth layers peak: Llama and Mistral peak at layer 16 of 32, while Qwen7B peaks at layer 21 of 28 and Qwen14B at layer 36 of 48.Postnorm joint scores are lower than mid-depth peaks; final-residual norm edits do not create the relation in separate cosine scores.Source: §5 · p. 8
- 0.77–0.85 joint AUC versus 0.54–0.60 late fusion: Chinese confirm repeats the pattern for multilingual models, but it vanishes for GPT-2, which was not trained on Chinese.Source: §5 · p. 8
6 One operator across independently trained models
Independently trained models share a graded relation rather than merely matching binary labels. Their errors also overlap, and a small reader can learn their consensus from unlabelled pairs.
6 One operator across independently trained models
- 0.855 average pairwise Spearman: six models’ PAWS item scores agree, with cross-family Qwen/Llama/Mistral agreement at 0.834.Within paraphrases or non-paraphrases alone, cross-family agreement remains 0.61–0.62.Source: §6 · p. 8
- 2.9–5.7× base rate: when one model errs on PAWS, another is more likely to err too, and averaging six outputs does not beat the best single model.The seven items all six models miss are three orders of magnitude more frequent than independent errors would predict; PAWS scores are 0.963 versus 0.970 and QQP scores 0.773 versus 0.778.Source: §6 · p. 8
- 0.89–0.93 Spearman: swapping sentence order barely changes scores, and all six directions order SNLI entailment categories monotonically, with entailment-versus-contradiction AUC 0.79–0.98.Source: §6 · p. 8
- 18 overlap-matched QQP items receive unanimous model disagreement with published labels; frozen-coordinate transfer to MRPC is partial at 0.48–0.61.Where the frozen coordinate transfers less, the models also agree less.Source: §6 · p. 8
- 0.959 PAWS confirm: a Qwen2.5-1.5B reader trained on about 10k unlabelled English pairs matches the 7B teacher’s 0.956 probe and reaches 0.758 on Chinese PAWS zero-shot.No gold pair label enters after the teachers’ coordinates are located.Source: §6 · p. 9
7 Scale, family, and a fossil that grows
The joint relation strengthens with scale but reaches an early ceiling in Qwen2.5; older model families reach similar joint-over-vector separation more slowly.
7 Scale, family, and a fossil that grows
- 0.916: Qwen2.5-1.5B base already reaches this joint score, compared with 0.947 instruct and 0.961 for 7B-Instruct, so instruction tuning is not required for the effect.Source: §7 · p. 9
- 0.789 (0.5B) → 0.916 (1.5B) → 0.937 (3B) → 0.941 (7B) → 0.950 (14B) → 0.938 (32B): Qwen2.5 joint AUC rises early and then largely saturates.At 32B, late fusion remains at 0.493.Source: §7 · p. 9
- 0.665/0.688/0.652/0.756: GPT-2 joint scores for small/medium/large/XL show a long plateau before an XL jump, while late fusion stays near chance.Other older lineages show plateau-and-jump patterns, including OPT’s 0.658–0.674 through 1.3B and 0.829 at 6.7B.Source: §7 · p. 9
8 The mouth is optional
A model’s joint state can expose identity even when its generated verdict does not. The split between computation and utterance varies by model family.
8 The mouth is optional
- 0.915, 0.938, 0.960, and 0.987 verdict AUC: Qwen base models’ zero-shot answers improve from 3B through 32B, while length-normalised PMI stays at 0.47–0.61.The 7B joint probe is 0.941, and the 7B-Instruct verdict reaches 0.915.Source: §8 · p. 9
- 0.614 and 0.606 verdict AUC: Llama 3 8B and Mistral 7B score far below their joint probes of 0.897 and 0.915; prompting and conditional surprisal do not recover the joint-state result.Older and smaller families show the same split, with verdicts at 0.47–0.61 against joint scores up to 0.893.Source: §8 · p. 9
9 What this is worth in the world
The trap-bank test shows that cosine can confidently select a meaning-changed twin, while a joint reader can certify candidates without replacing the frozen retrieval index. On Quora wording twins, the paper also reports strong cosine retrieval and abstention metrics.
9 What this is worth in the world
- 0.994 recall@10, 0.862 cosine top-1, and 0.985 in/out abstention AUC are reported for the Quora wording-twin bank.The queries are held-out duplicate partners in a bank where restated questions still share words.Source: §9 · p. 9
- All systems receive the same cosine top-10, with thresholds set on a random 30% of queries and results reported on the rest.The trap bank contains 1,289 in-KB queries and 300 out-of-KB queries whose meaning-changed twin is in the KB.Source: Table 2 · p. 10
- 0.900 fused: the consensus-trained 1.5B reader reproduces the trap-bank result with no gold labels and no 7B at inference.On trapped queries, cosine top-1 is 0.468; confident wrong answers fall from 0.284 to 0.096 when the reader is fused with cosine.Source: §9 · p. 10
- 0.83: the frozen PAWS direction certifies the same cosine top-10 on trapped queries with zero in-domain labels.Fused with cosine, it reaches 0.90 overall, matching the prompted 7B verdict of 0.903.Source: §9 · p. 10
- The proposed division of labor is to retrieve with embeddings and certify identity with a joint pass, while BGE-base STS-B Spearman stays 0.895.Source: §9 · p. 10
10 Limitations
The stated scope is PAWS-style paraphrase under overlap matching; the reported cross-architecture pattern does not broaden that evaluation scope.
10 Limitations
- The evidence is scoped to PAWS-style paraphrase under overlap matching, so broader paraphrase settings remain outside the stated scope.Source: §10 · p. 10
11 Conclusion
The conclusion frames meaning identity as computed in a joint pass rather than residing in frozen independent embeddings. It reports that this pattern spans architectures and can be distilled into a reader that repairs a cosine FAQ stack.
11 Conclusion
- Meaning identity on overlap-matched PAWS-X is a joint-pass computation, not a property of frozen independent embeddings.Source: §11 · p. 10
- The joint-versus-independent result spans causal, encoder, and encoder–decoder families, survives nonce substitution, and is shared across training houses.Source: §11 · p. 10
- The relation distills into a cheap reader that repairs a cosine FAQ stack.Source: §11 · p. 10