PaperlayerView exact revision on arXiv

Source-linked AI summary

A Better Metric Will Not Save You: What You Compare Beats How You Compare It

Noureddine Khaous, Guillaume Wisniewski

arXiv:2610.13476v1cs.CL

TL;DR

Representational similarity measures leave the set of conditions fixed, although its inputs and grouping are decided before the measures operate. The paper tests those choices against vowel acoustics across languages and representations, finding that which categories are compared can move the measured gap more than the effect itself, while how many categories are compared has no measurable effect.

  • Problem

    Representational similarity measures take their condition set as given, despite the set being determined by prior decisions about which inputs enter and how they are grouped.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    The paper evaluates condition-set choices against vowel acoustics, an arrangement fixed independently of the systems compared, across fifteen languages and twenty-seven representations.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    Changing which categories are compared shifts the gap by 0.146 on average against an effect of 0.0545, while changing the count from five to twenty-eight moves it by only 0.005.

    Source: §1 · p. 2

  • Takeaways & Limitations

    In this setting, composition of the condition set moves the pre-training gap more than the gap itself, whereas the three measures rank twenty-seven representations alike.

    Source: §5 · p. 9

    The reported magnitudes are established for vowel geometry in multilingual speech, while two results rely on English, the only sampled language explored widely.

    Source: §5 · p. 9

Abstract

from arXiv · show

Representational similarity measures (RSA, CKA, Procrustes) are chosen for the transformations they treat as irrelevant, and all of them take the set of conditions as given. That set is fixed by two decisions, which inputs are admitted and how they are grouped, both taken before the matrix a measure operates on exists: no invariance property reaches them. We measure what they are worth where the arrangement the conditions ought to have is known independently of the systems compared, the acoustic organisation of vowels, across fifteen languages and twenty-seven representations. Changing which categories are compared, at a fixed number of them, moves the gap between two representations by more than twice the effect being measured, while changing how many moves it not at all. In this setting rankings of models are identified under these choices, and rankings across languages are not.

1 INTRODUCTION

Representational similarity measures compare systems over a fixed set of conditions, but the choices that form that set precede the matrices and evade measure-level invariance. Using vowel acoustics as an independent reference, the paper finds that category composition can shift a comparison more than the effect being measured.

Motivation

  • Representational similarity claims inform choices of architectures and assessments of which cognitive accounts remain viable.The measures themselves can disagree on the same data, and the debate over which to choose has remained open.Source: §1 · pp. 1–2

Problem

  • Representational similarity measures compare pairwise relations among a fixed set of conditions, determined by which inputs enter and how they are grouped.These choices are made before the N × N matrix exists, so invariance properties of the measure do not reach them.Source: §1 · pp. 1–2

Approach

  • The paper measures condition-set choices against vowel acoustics, whose geometry is recovered independently of the systems being compared.The acoustic organisation of vowels varies by language, making it possible to ask the question across languages.Source: §1 · p. 2

Motivation

  • Condition-set choices are often constrained but underdetermined, so they can be mistaken for facts about the data.The paper’s vowel comparison requires decisions about which audio stretches count as instances and which categories those instances form.Source: §1 · p. 2

Main finding

  • Choosing which vowel categories to compare shifts the gap by 0.146 on average, versus the 0.0545 effect being measured.Changing the number of categories from five to twenty-eight moves the gap by only 0.005, around its standard error.Source: §1 · p. 2

Contribution

  • The paper treats the condition set as an object of measurement and asks which comparative claims remain identified when its choices vary.Its contributions distinguish effects that alter correctness from those that affect only stability.Source: §1 · p. 2

2 BACKGROUND: MEASURING REPRESENTATIONAL SIMILARITY

Representational similarity measures compare the internal arrangements of conditions rather than matching individual dimensions across systems. Their invariances concern the resulting matrices, while the condition set supplied to those measures remains a prior analytic choice.

Condition sets

  • When each row is a single input, only the choice of admitted inputs applies; when rows aggregate, both admission and grouping apply.The applicable choices therefore depend on what counts as a condition in the analysis tradition.Source: §2 · p. 3

Measures

  • RSA compares distance matrices, linear CKA aligns condition Gram matrices, and Procrustes minimises vector mismatch over rotations.RSA leaves the distance and triangle-comparison statistic open; Procrustes requires a common dimension whose projection rank is not supplied by its definition.Source: §2 · p. 3

Invariance

  • The measures index objects by conditions rather than dimensions, yielding permutation invariance and, for Euclidean proximity, orthogonal-rotation invariance.This discards coordinate space and leaves downstream comparisons dependent on how the condition arrangement was built.Source: §2 · p. 3

Prior work

  • Prior work has debated representational-similarity measures through invariance properties and internal comparisons, while experiments show measures can disagree.The literature’s proposals concern what a measure does to the N × N matrix, not the condition set it receives.Source: §2 · p. 3

Condition sets

  • The condition set is fixed before these measures receive it, leaving input admission and, where rows aggregate, grouping outside their invariance properties.A row may represent a single input, repeated-trial summary, or category pooled across inputs; model-to-brain comparisons may pool the two sides differently.Source: §2 · p. 3

3 AN INSTRUMENT WITH A KNOWN ANSWER

The paper uses vowel acoustics as an external reference with a known arrangement, then anchors comparisons to a scale defined by controls. Its protocol varies required but underdetermined choices and distinguishes shifts shared across systems from shifts that differ between them.

Reference geometry

  • Vowel categories occupy stable regions of a two-dimensional acoustic plane, providing a geometry against which each system’s arrangement can be compared.The coordinates are the first two formants, F1 and F2, recovered from the waveform by standard signal processing.Source: §3 · p. 4
  • The reference is a measurement fixed before any model is consulted, not a ground truth or a claim about what speech representations ought to encode.Its alignment is produced by a trained model outside the evaluated systems, so alignment errors are common to them rather than favouring one.Source: §3 · p. 4

Data and systems

  • The study uses fifteen FLEURS languages and evaluates twenty-seven representations, including trained systems and controls spanning the comparison scale.The sampled languages span inventories of five to twenty-eight categories; absent speaker labels constrain what can be controlled for.Source: §3 · p. 4

Protocol

  • Token frames are mean-pooled, then category tokens are summarised by a representation-side mean and reference-side coordinate-wise median before RSA comparison.The default uses cosine distance for representations, Euclidean distance for the reference, and Spearman’s ρ between vectorised distance triangles.Source: §3 · p. 4

Anchored scale

  • Effects are expressed as percentages of the interval between an untrained architecture and exact recovery of the geometry.The two endpoints are controls, avoiding a scale defined by the model effect being judged.Source: §3 · p. 5
  • RSA’s interval is the widest of the three at 0.49755, compared with 0.3331 and 0.1742 for the other measures.The paper chooses it for resolution, while noting that reading each measure on its own interval returns the pre-training gap with the same sign and order of magnitude.Source: §3 · p. 5

Reference reliability

  • Reference split-half reliability ranges from 0.968 to 1.000 across languages, with a median of 0.997, but stability does not establish validity.A consistently wrong octave jump by the formant tracker could reproduce in both halves and leave the reliability ceiling untouched.Source: §3 · p. 5

Choices and effects

  • A bifurcation is a required, unconventioned, rarely reported choice that affects compared systems asymmetrically.The analysis separates common score shifts from differential shifts, because only the latter bears on rankings.Source: §3 · p. 5

4 RESULTS

Across the measured choices, changing the condition set can move representation comparisons substantially, while the measures agree more on broad rankings. The effects vary by choice and inventory size, and some rankings remain unstable.

4.2 HOW THE INPUTS ARE GROUPED

  • 0.146 average composition-driven gap separation exceeds the 0.0545 effect, while changing set size from five to twenty-eight moves it by less than its standard error.The figure expresses each choice’s displacement as a percentage of the anchored interval, with the vertical line marking the pre-training gap.Source: Figure 1 · p. 6, §4.2 · p. 7

4.1 WHAT THE MEASURES DECIDE, AND WHAT THEY LEAVE OPEN

  • The three measures rank twenty-seven representations alike and preserve the pre-training gap’s sign and order of magnitude, but disagree more among trained and spectral representations.Varying distance conventions changes the verdict in four of five rank-by-domain configurations, in both directions, with interacting factors.Source: §4.1 · p. 6

4.2 HOW THE INPUTS ARE GROUPED

  • Merging twenty-eight English categories into twelve raises scores by +0.0768 on average, benefiting the six weakest representations more than the six strongest and narrowing one model gap from 0.317 to 0.224.The hand-built twelve-category English inventory falls at percentile 71.8 among random subsets of its size.Source: §4.2 · p. 7

4.3 WHICH INPUTS ARE ADMITTED

  • In Polish, the only sampled language where the spectral baseline leads, changing the duration threshold moves the gap from -0.047 to -0.002; the proposed bias diagnostic does not predict differential effects.Across fifteen languages, measured bias and differential effect correlate +0.118 (p = 0.676).Source: §4.3 · pp. 7–8

4.4 WHAT THE ANALYST DOES DOWNSTREAM

  • 0 of 32 paired comparisons survive the layer-selection protocol change, whose inflation is 5.2% of the anchored interval against a median gap of 0.0564.On dSprites, with an exact reference from known generative factors, a baseline contests five of twelve comparisons at matched relative depth.Source: §4.4 · p. 8

4.5 WHAT SURVIVES

  • Composition alone has an interval separating it from the other measured displacements; five other choices are comparable, while aggregation is negligible.The composition estimate lacks a multilingual interval because only one language has twenty-eight categories; standardising the reference over the full inventory yields +12.7% instead of +10.7%.Source: §4.5 · p. 8
  • Above +0.922, model rankings correlate across both undeclared choices, but at five categories the trained-representation ranking correlation falls to +0.313 between draws.The paper concludes that model rankings are identified under the choices in this setting, whereas rankings across languages are not.Source: §4.5 · pp. 8–9

5 CONCLUSION

The conclusion is that condition-set choices precede the operations of representational similarity measures and can shape their comparisons. The authors recommend reporting those choices and bound the measured magnitudes to vowel geometry in multilingual speech.

5 CONCLUSION

  • A measure’s invariances apply after the N × N matrix exists, so they cannot reach decisions that fix the condition set beforehand.The argument concerns the ordering of analytic decisions, not a rejection of the invariance literature.Source: §5 · p. 9
  • Comparative claims should report the condition set category by category, disclose and sweep the input-admission rule, and match system depth or state otherwise.The authors describe their own protocol record as a base rate, not evidence of rigour; preregistration, a bug journal, and a test suite still left defects.Source: §5 · p. 9
  • The measured magnitudes are established for vowel geometry in multilingual speech, while the structural claim is not confined to that setting.Two results use English; only layer-selection was tested in a second modality, and at least two further bifurcations remain unexamined.Source: §5 · p. 9
  • An external account of how conditions ought to be arranged can distinguish a wrong conclusion from a merely different one; the authors call for more domains with such independent structure.They cite a layer-selection artefact reproduced, and larger, where generative factors provide the reference exactly.Source: §5 · p. 9

REPRODUCIBILITY STATEMENT

Reported values are generated from stored results through a validation pipeline, and the extraction, analysis, figure, and regeneration code is included in supplementary material.

REPRODUCIBILITY STATEMENT

  • The pipeline generates values from stored result files and binds each value to an artefact and checksum.It rejects conflicting bindings, quantities sourced from different artefacts, and percentages that cannot be recomputed from their operands.Source: REPRODUCIBILITY STATEMENT · p. 10
  • After the last recorded defect, the pipeline was frozen and every value regenerated; the largest correction changed the anchored interval in its fifth decimal without changing a conclusion.A few exceptions are fixed protocol constants and figures from exploratory runs whose scripts write no result file.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The supplementary material includes extraction, analysis, figure, and value-regeneration scripts; the narrower sweep uses eight representations because stored representations predate duration recording.Extending that sweep requires re-extracting nineteen models over fifteen languages, estimated at about three GPU-days.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

The study analyses public corpora and checkpoints without collecting new human-participant data. Its language comparisons also bear on equity-related claims, which the authors caution are not identified under undeclared choices.

ETHICS STATEMENT

  • The work analyses public corpora and public model checkpoints and collects no new data from human participants.FLEURS is used as published, without added annotation.Source: ETHICS STATEMENT · p. 10
  • Because FLEURS carries no speaker identifiers, the study’s artefacts bound what could be inferred about any individual.Source: ETHICS STATEMENT · p. 10
  • The authors caution that claims about multilingual models serving languages equally or adequately are fragile because cross-language comparisons are not identified under undeclared choices.They recommend reporting the condition set and the rule admitting inputs.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

The authors describe limited language-model assistance, while the appendix states how linear CKA and orthogonal Procrustes compare representations.

AI USE STATEMENT

  • The language model assisted with drafting, revision, consistency checks, and code preparation, but not research ideas or hypotheses; all reported numbers come from the analysis pipeline.The authors take responsibility for the paper’s content.Source: AI USE STATEMENT · p. 10

A THE TWO MEASURES STATED IN FULL

B EXPERIMENTAL PROTOCOL

The protocol builds category-level representations and an independent vowel reference, then explores defensible condition sets and reports effects on a common displacement scale.

B EXPERIMENTAL PROTOCOL

  • Tokens are mean-pooled into vectors and grouped by phone label, while reference categories use coordinate-wise median formants to resist gross estimation errors.The coordinate-wise median commutes with monotone per-coordinate transformations; its order relative to Bark warping and standardisation is immaterial.Source: B EXPERIMENTAL PROTOCOL · p. 12
  • The analysis samples subsets of categories that each language actually distinguishes, rather than arbitrary partitions that could create unattested vowel centroids.For merging, draws match published inventory category-size profiles and remain within attested dispersion.Source: B EXPERIMENTAL PROTOCOL · p. 12
  • The vowel reference is recoverable directly from recordings without models or annotations, making it independent of the systems being evaluated.The reference is described as reproducible from a microphone and about an hour of recordings.Source: B EXPERIMENTAL PROTOCOL · p. 13
  • The Spanish noise ceiling of 1.000000 reflects identical ordering of only ten distances, while split-half reliability measures reliability rather than validity.The Spanish ceiling is not used as a denominator.Source: B EXPERIMENTAL PROTOCOL · p. 13
  • Table cells generally use the default configuration, median per-language scores across fifteen languages, and percentile-bootstrap intervals; unavailable rank–cardinality combinations are left empty.A projection rank d requires k ≥ d + 1 conditions, so rank and cardinality cannot be crossed freely.Source: B EXPERIMENTAL PROTOCOL · p. 13

D DISPERSION IS NOT DISPLACEMENT

  • Table 7 places each choice’s native effect quantity alongside its displacement of the SSL–spectral gap, expressing effects on a common anchored scale.The table’s value column intentionally mixes units because each effect was first reported in its analysis-specific quantity.Source: D DISPERSION IS NOT DISPLACEMENT · p. 14, Table 7 · p. 14

E THE RATIO, AND THE CONVENTIONS OF OUR OWN PIPELINE

The paper examines pipeline conventions, alternative comparison approaches, language and representation choices, inventory construction, and a vision-model layer-selection test.

E THE RATIO, AND THE CONVENTIONS OF OUR OWN PIPELINE

  • Changing distance convention can alter the RSA-versus-CKA slope comparison, while linear CKA’s slope remains +0.00722 under both conventions.The passage attributes this difference to RSA’s free representation-side distance and CKA’s distance fixed by the linear kernel.Source: E THE RATIO, AND THE CONVENTIONS OF… · p. 14, Table 8 · p. 15

F ALTERNATIVES TO STRUCTURAL COMPARISON

  • Probing measures decodability and depends on the mapping class, while behavioral error consistency can remain low even when system accuracies match closely.The passage also notes that systems with different internal geometries can have similar error profiles.Source: F ALTERNATIVES TO STRUCTURAL COMPARISON · p. 15

G LANGUAGES

  • The sample uses fifteen of FLEURS’ 102 languages, spanning eleven families and vowel inventories from five to twenty-eight categories.The selection varies expectedly relevant dimensions including inventory size, tone or pitch accent, and language family.Source: G LANGUAGES · p. 15

H REPRESENTATIONS

  • The nine random-weight controls span three architectures and three seeds; each differs from its trained counterpart by training alone.The anchored-scale floor pools these controls and indicates where an untrained network of this family falls, not seed-to-seed variation within an architecture.Source: H REPRESENTATIONS · p. 16

I WHY NO INVENTORY IS CANONICAL

  • Vowel inventories are analytic choices rather than observations, and the English inventory’s 28 categories and project-built 12-category merge both describe the same recordings.The passage reports that merging raises every score, with the largest rise for the weakest representations.Source: I WHY NO INVENTORY IS CANONICAL · pp. 17–18

J THE LAYER-SELECTION ARTEFACT IN VISION

  • On dSprites, a fixed relative-depth layer rule lets a baseline contest five of twelve model pairs, and raw pixels rank ahead of all three trained vision models.The composition of the condition set was not tested in vision; the reported composition result remains about speech.Source: J THE LAYER-SELECTION ARTEFACT IN VISION · p. 19

K WHAT THIS PAPER DOES NOT ESTABLISH

The findings have bounded reach: some results are English-only, and the study does not exhaust the choices or comparisons its instrument could examine.

K WHAT THIS PAPER DOES NOT ESTABLISH

L THE RECORD: PREDICTIONS AND DEFECTS

The authors' audit records substantial prediction failure alongside pipeline defects that produced plausible outputs or escaped checks that appeared to run.

L THE RECORD: PREDICTIONS AND DEFECTS

  • Five of ten registered predictions were refuted outright, four partly confirmed, and one untested; two refutations had correct premises but incorrect conclusions.Source: L THE RECORD: PREDICTIONS AND DEFECTS · p. 20
  • Matching aggregation estimators widened rather than narrowed the gap, and diagnostics twice measured a property correctly but failed to predict the bifurcation's effect.A spurious correlation was caught by testing at fixed n, where the protocol prediction engaged and returned nothing.Source: L THE RECORD: PREDICTIONS AND DEFECTS · pp. 20–21
  • The authors kept two supervised encoders in the preregistered self-supervised panel after discovering the classification, and reported sensitivity instead of redefining it.Source: L THE RECORD: PREDICTIONS AND DEFECTS · p. 21
  • A pooling asymmetry inflated an early effect fourfold, while single-precision cancellation perturbed the ordering of tiny distances in some representations.At reported layers, the precision displacement was at most 0.0066 and changed the default layer for one model–language pair.Source: L THE RECORD: PREDICTIONS AND DEFECTS · p. 21
  • Missing token provenance blocked the duration sweep beyond 8 of 27 representations, while absent checkpoint revisions weakened reproducibility claims.Both kinds of information were available when computing but were not stored with the results.Source: L THE RECORD: PREDICTIONS AND DEFECTS · pp. 21–22
  • A percentile shifted by seven points across runs because earlier analysis consumed different amounts of a shared random stream, changing later draws.Source: L THE RECORD: PREDICTIONS AND DEFECTS · p. 22
  • Three controls failed to check what they were meant to check; one missed an over-wide table because log parentheses misattributed its overflow.The build now fails on overflow it cannot attribute.Source: L THE RECORD: PREDICTIONS AND DEFECTS · p. 22

M PREREGISTRATIONS

  • The preregistration record is only partly externally checkable: two predictions were in an unversioned plan, and the verdict journal entered version control later.Source: M PREREGISTRATIONS · p. 22

N A CHECKLIST

The checklist turns the paper's findings and audit into reporting and validation practices for comparative analyses.

N A CHECKLIST

  • Report the condition set category by category; four coverage statistics capture only a tenth of what is at issue.Source: N A CHECKLIST · p. 22
  • Report and sweep the input-inclusion rule: it moved a comparison from -0.047 to -0.002 in one language, and no cheap diagnostic predicted its effect.Source: N A CHECKLIST · p. 22
  • Compare systems at matched depth or disclose otherwise; best-layer reporting accounts for 46% of the margin through unequal numbers of layer draws.Source: N A CHECKLIST · p. 22
  • Check estimator resolution as condition sets shrink, and report a noise ceiling while noting that systematic measurement error can evade split-half reliability.Source: N A CHECKLIST · p. 23
  • Compute distances at comparison precision, and record token identity and checkpoint revision when results are produced because that provenance cannot be recovered later.Source: N A CHECKLIST · p. 23
  • Before reporting a correlation, verify both terms vary within the analysis unit; make controls fail when they cannot check, since checklists are no guarantee.Source: N A CHECKLIST · p. 23
Loading 2610.13476v1…