A Flow Matching Framework for Neural Representational Dissimilarity

Zeyuan Ye; Xue-Xin Wei

Source-linked AI summary

A Flow Matching Framework for Neural Representational Dissimilarity

Zeyuan Ye, Xue-Xin Wei

arXiv:2609.31544v1Sep 25, 2026cs.AIcs.ITcs.LGmath.IT

TL;DR

Neural response distributions can be compared with diverse distance metrics, each reflecting different properties, while a common framework and estimator are lacking. This paper uses flow matching to derive metrics as Jeffreys divergences under different velocity constraints, reporting advantages for complex distributions and continuous variables while enabling principled metric design.

Problem
Different neural dissimilarity metrics capture different properties and use separate assumptions, motivating a unified framework that can clarify their interpretation and estimation.
Source: §1 · pp. 1–2
Method
The framework fits target response distributions by flow matching under chosen velocity constraints and defines their representational distance as a Jeffreys divergence.
Source: §1 · p. 2, §3 · p. 3
Results
Common distance metrics arise as Jeffreys divergences under different velocity constraints, and the framework has empirical advantages for complex distributions and continuous variables.
Source: §1 · p. 2, §4 · p. 5, §6 · p. 9
Takeaways & Limitations
Choosing a source distribution and velocity constraint provides a principled way to design new representational distance metrics.
Source: §4 · p. 5
The theoretical correspondences hold at global population optima that neural-network training may not reach, and validation has focused on visual-cortex recordings.
Source: §6 · p. 9

Abstract

from arXiv · show

Neural representational dissimilarity quantifies differences between neural response distributions, and is essential for comparing neural codes across stimuli, brain areas, tasks, and models. Commonly used distance metrics involve different assumptions and are estimated with separate methods. Here, we show that a variety of distance metrics can be unified under a flow matching framework developed in deep generative models. That is, these distances arise as Jeffreys divergences under different velocity constraints. We find that flow matching has advantages for estimating distances involving complicated distributions and continuous variables. Furthermore, this framework enables the design of new distance metrics in a principled way. Together, flow matching provides a unified approach for understanding, estimating, and designing neural representational dissimilarity metrics.

1 Introduction

Neural representations are studied by comparing response distributions across conditions, but different dissimilarity metrics capture different distribution properties and rely on separate assumptions. The paper proposes flow matching as a framework that unifies these metrics and supports principled design of new ones.

1 Introduction

Representational similarity analysis compares neural population response distributions across conditions using pairwise distances.It is used to study perception and task performance and to compare representations across species and computational models.Source: §1 · p. 2
Different metrics reveal different properties of neural response distributions, so no single distance is best for every purpose.Examples include mean-pattern correlation, Euclidean separation of means, covariance-aware Mahalanobis distance, full-distribution KL divergence, and local Fisher information.Source: §1 · p. 2
A unified framework could clarify metrics’ statistical assumptions, guide new metric design, and let estimator improvements benefit multiple metrics.These are the stated motivations for seeking a common framework rather than separate estimation methods.Source: §1 · p. 2
The paper unifies varied distance metrics with flow matching, expressing them as Jeffreys divergences under different velocity constraints.The framework is also presented as a way to estimate distances for complex distributions and continuous variables and to design new metrics.Source: §1 · p. 2

2 Background and Related Work

Prior work uses representational dissimilarity matrices to summarize pairwise separations, while Fisher information characterizes sensitivity to continuous conditions. The paper situates flow matching as a common framework for these traditionally separate approaches.

Representational dissimilarity analysis

Representational dissimilarity matrices summarize neural, behavioral, or model representations through pairwise condition separations.Common entries include correlation, Euclidean, Mahalanobis, and cross-validated Mahalanobis distances, which differ in their treatment of scale, noise covariance, and finite-sample bias.Source: §2 · p. 2

Fisher information estimation

Fisher information measures response-distribution change with a continuous condition and locally quantifies discriminability.With repeated nearby stimulus values it can be estimated from tuning-curve derivatives and response covariance or decoder-based estimators; continuous and naturalistic settings lack exact repeats, and Gaussian processes scale poorly.Source: §2 · p. 2

Continuous normalizing flows and flow matching

Flow matching trains a transport velocity by regression along a specified probability path, avoiding the likelihood-based ODE and divergence calculations used in earlier continuous-flow training.The framework brings Fisher information estimation together with other distance metrics traditionally treated separately.Source: §2 · pp. 2–3

3 A flow matching framework for computing representational distance

The framework fits condition-specific response distributions by transporting a shared source through a learned velocity field. The chosen source and velocity class determine which target properties are represented; the resulting fitted distributions are compared with a Jeffreys divergence.

3 A flow matching framework for computing representational distance

Flow matching learns a time-dependent velocity field that transports samples from a simple source distribution toward each condition’s target response distribution.The flow coordinate t is artificial rather than physical time.Source: §3 · p. 3
The framework’s three steps are to choose a source and velocity class, learn the field to approximate target distributions, then compute their Jeffreys divergence.The learned transport maps the source to fitted condition-specific distributions that approximate the target responses.Source: §3 · pp. 3–4
The source is typically shared across conditions, and the prescribed interpolation schedules control source and target contributions along the flow path.The path begins at the source sample and ends at the recorded response.Source: §3 · p. 3
Choosing a source distribution and velocity class determines which transformations of the target distributions can be represented and thus which distance is induced.A state-independent constant velocity restricts transport to translation, while an unconstrained neural velocity allows flexible nonlinear transformations.Source: §3 · p. 3
Training draws source and observed-response samples, connects them along a predefined path, and regresses a neural network onto the target instantaneous velocity.The same procedure applies to continuous conditions by sampling the condition variable and replacing the categorical condition in the training setup.Source: §3 · pp. 3–4
With a Gaussian source and affine velocity, deterministic ODEs for fitted means and covariances allow the Jeffreys divergence to be computed.For other cases, the divergence is estimated by Monte Carlo using samples integrated through the sampling ODE and their log densities.Source: §3 · p. 4

4 Flow matching with different velocity constraints lead to different representational metrics

The paper shows that commonly used representational distances arise as Jeffreys divergences under different velocity constraints. It also demonstrates a principled way to design a new metric by choosing a source template and restricting its transformations.

4 Flow matching with different velocity constraints lead to different representational metrics

Common representational distance metrics arise as Jeffreys divergences under different velocity classes.The proof strategy finds each constrained field’s population-loss minimizer, determines its induced distribution, and evaluates the Jeffreys divergence between condition-specific approximations.Source: §4 · pp. 4–5
A constant velocity yields squared Euclidean distance between target means, an affine velocity yields Gaussian Jeffreys divergence, and an unconstrained field yields full Jeffreys divergence in the population limit.The constant field shifts the source mean without changing covariance; the affine field produces a moment-matched Gaussian approximation, while the unconstrained field recovers the target distribution.Source: §4 · p. 5
A new metric can be designed by using a source distribution as a geometric template and restricting its transformations; the paper demonstrates similarity-transform constraints.This metric may be useful when target geometry is known a priori or when a particular geometric property is to be emphasized.Source: §4 · p. 5

5 Applications

The applications test flow matching across categorical RDMs, Fisher information, and time-resolved distances, using simulations and neural recordings. The results show how estimates vary with velocity constraints, distribution complexity, and continuous conditions.

5.1 Flow matching recovers multiple distance metrics on a synthetic categorical dataset

For the five-condition Gaussian dataset, RDM-estimate correlations improved with sample size across all six metrics, and estimates closely matched ground truth.On this simple dataset, flow matching showed no accuracy advantage over specialized estimators.Source: §5.1 · pp. 5–6, Figure 3 · p. 6

5.2 Flow matching estimates Fisher information in simulations and neural data

On Gaussian toy data, affine flow tracked ground-truth linear Fisher information; on mixture data, only unconstrained flow successfully estimated full Fisher information.GKR and the Wishart process errors increased with response dimension on the Gaussian dataset and failed on mixtures because they assume Gaussian distributions.Source: §5.2 · p. 6, Figure 4 · p. 7
Across six mouse sessions, unconstrained flow had the highest held-out log likelihood overall, while affine-flow linear and unconstrained-flow full Fisher estimates both peaked near 90°.The analysis used PCA-reduced responses; the orientation peak suggests greater discriminability of horizontal visual patterns.Source: §5.2 · p. 7, Figure 5 · p. 7

5.3 Estimating time-resolved distances using flow matching

In OU-trial simulations, affine flow outperformed conventional methods for Gaussian Jeffreys divergence, while only unconstrained flow outperformed them for mixture Jeffreys divergence.The Gaussian divergence permits different condition covariances; the full divergence makes no Gaussian assumption.Source: §5.3 · p. 8, Figure 6 · p. 8
Across five temporal frequencies in Neuropixels recordings, distances rose after stimulus onset and fell after offset; faster gratings had smaller distances, and opposite directions were closer than orthogonal ones.These estimates were based on 32 recording sessions.Source: §5.3 · pp. 8–9, Figure 7 · p. 9
RDM correlations averaged 0.73 during stimulus presentation, versus 0.055 before and 0.25 after; GKR tended to oversmooth and produce spuriously high pre-stimulus correlations.The authors interpret the results as relatively stable representational geometry during stimulus presentation despite changes in manifold scale.Source: §5.3 · p. 9

6 Discussion and Conclusion

The authors conclude that flow matching unifies neural representational dissimilarity metrics and can estimate complex distributions conditioned on continuous variables. They qualify the conclusion by noting training and validation limits.

6 Discussion and Conclusion

Flow matching expresses neural dissimilarity metrics as Jeffreys divergences under different velocity-field constraints, providing a common estimator and a principled way to design metrics.Theoretical correspondences hold at global population optima, which neural-network training may not reach.Source: §6 · p. 9
The framework’s validation focused on visual-cortex recordings; extending it to other brain systems, measurement techniques, and deep neural network models remains future work.The authors specifically mention fMRI and EEG as additional measurement techniques.Source: §6 · p. 9

AI use statement

AI-assisted tools supported language editing and manuscript organization under author direction; the authors reviewed and take responsibility for the final scientific content.Source: AI use statement · p. 10

Ethics statement

The study analyzed synthetic data and previously collected animal recordings, with no new animal or human-subject experiments.Source: Ethics statement · p. 10

Reproducibility statement

Supplementary materials provide proofs, estimator specifications, dataset details, hyperparameter searches, and additional results; code is available in the FlowRDM repository.The real-data experiments use recordings described by Stringer et al. [2021] and Allen Brain Observatory datasets.Source: Reproducibility statement · p. 10

Supplementary Material Contents

Flow-matching training under different velocity constraints yields Jeffreys divergences corresponding to existing representational distance metrics. The appendix states the assumptions and develops affine-flow results that permit deterministic evaluation for Gaussian endpoints.

A Mathematical Proofs of the Metric Correspondences under flow matching

Different velocity constraints in flow-matching training yield Jeffreys divergences corresponding to existing distance metrics; optional log-likelihood fine-tuning recovers the same metrics.The appendix states these metric correspondences as population results.Source: A Mathematical Proofs of the Metric… · p. 14

A.1 Assumptions and notation

The framework treats representational distance as symmetric and nonnegative, while requiring neither Gaussian response distributions nor triangle-inequality compliance.The stated setup also assumes finite second moments, invertible covariance matrices, and specified differentiable schedules.Source: §A.1 · p. 15

A.2 Affine-flow solution

An affine flow maps a Gaussian source through an affine ODE, preserving Gaussianity at every time point.The proof derives the solution from the ODE and initial condition, then uses uniqueness of the linear ODE.Source: §A.2 · p. 15

A.3 Deterministic Jeffreys evaluation for affine Gaussian flows

For affine fitted velocities, integrating the mean and covariance dynamics gives Gaussian endpoint distributions whose pairwise Jeffreys divergence has a closed form.Computing it from endpoint moments requires no generated samples or sample-wise log-density evaluations, and has no Monte Carlo sampling error.Source: §A.3 · p. 15

A.4 Proofs of velocity-to-distance metric correspondences

The proofs establish how velocity constraints determine the recovered distributional distances, including correspondences to distances between condition means, Jeffreys divergence, and local Fisher information. They also characterize constrained affine or similarity flows and show that optional NLL fine-tuning preserves the velocity constraint and recovers the distance correspondences at population optima.

A.4 Proofs of velocity-to-distance metric correspondences

Translation-constrained flows recover squared Euclidean distance between condition means, even when the response distributions are non-Gaussian.The fitted endpoint distributions are Gaussian because the Gaussian source is translated.Source: §A.4 · p. 16
Fixed-norm translations induce 2r² times cosine distance between condition means by retaining their directions but removing mean-magnitude information.If a condition mean is zero, the constrained optimum is nonunique and cosine distance is undefined for that condition.Source: §A.4 · pp. 16–17
Centered fixed-norm translations induce 2r² times Pearson correlation distance between condition means.When a centered condition mean is zero, the constrained optimum is nonunique and correlation distance is undefined for that condition.Source: §A.4 · p. 17
A shared centered affine flow induces squared Mahalanobis distance based on pooled within-condition covariance, using only response means and covariances.The response distributions need not be Gaussian.Source: §A.4 · pp. 18–19
Condition-specific centered affine flows yield Gaussian endpoint distributions and their induced Gaussian Jeffreys divergence, reducing to squared Mahalanobis distance when the condition covariances are equal.The equal-covariance result uses the shared covariance Σ.Source: §A.4 · pp. 19–20
Stimulus-dependent affine flows yield Gaussian Fisher information, while averaging local affine coefficients yields linear Fisher information.The Gaussian Fisher information includes mean-difference and covariance contributions; the averaged-coefficient construction gives the linear Fisher information result.Source: §A.4 · pp. 20–21
An unconstrained population flow recovers the target distributions, so its induced distance is true Jeffreys divergence and its local expansion gives full Fisher information [Lipman et al., 2023].For a regular continuous parameter family, the divergence expands as h⊤J(θ)h + o(∥h∥²).Source: §A.4 · p. 22
The map has the form T(z) = sRz + u, where s > 0, R ∈ SO(d), and u ∈ Rd.Source: §A.4 · p. 22
The constrained flow induces T(z)=sRz+u, with R in SO(d), excluding shear and nonlinear deformation.The proof establishes that the rotation remains orthogonal with determinant +1 and that the remaining trajectory term is a translation.Source: §A.4 · pp. 22–23

B.1 Optional negative-log-likelihood fine-tuning

At population optima, optimizing NLL recovers the distance correspondences in Table 1.Source: §B.1 · p. 23
Optional NLL fine-tuning uses change-of-variables densities and backward integration to raise density on recorded responses while keeping velocity constrained to V.Flow matching’s regression objective is fast to train but does not directly maximize likelihood; fine-tuning improves its finite-sample approximation to pc.Source: §B.1 · p. 23

B.2 Likelihood-based proofs

The likelihood-based proofs show how constrained flow families induce familiar neural distances, with corresponding results for Gaussian approximations and continuous-variable Fisher information.

B.2 Likelihood-based proofs

Translation flows fit condition-specific unit-covariance Gaussians, inducing squared Euclidean, cosine, or Pearson correlation distances under the corresponding constraints.The results assume finite second moments; the cosine and correlation constraints use a fixed norm and a centered sphere, respectively.Source: §B.2 · p. 24
When the required mean vector is zero, the constrained optimum is nonunique, matching the domain restriction of cosine or correlation distance.The response distributions themselves need not be Gaussian.Source: §B.2 · p. 24
Likelihood-trained affine flows with shared covariance induce a pooled-covariance Mahalanobis distance, while condition-specific covariances yield Gaussian Jeffreys divergence.These results assume positive-definite response covariances and describe population optima.Source: §B.2 · pp. 23–25
For continuous scalar conditions, pointwise likelihood-trained affine flows recover Gaussian Fisher information, while a shared local covariance yields linear Fisher information.The derivation assumes smooth means and covariances and pointwise realizability; finite parameterizations that couple distant conditions may not attain separate moment projections.Source: §B.2 · p. 25

C A new distance metric that preserves the geometry of a template

The section constructs a geometry-preserving distance by choosing a template source distribution and restricting its transformations, then demonstrates the metric on synthetic square data. It also documents estimator and dataset procedures used in the experiments.

C A new distance metric that preserves the geometry of a template

A flow-matching metric can be designed by choosing a source distribution and permitted velocity class; the example preserves a prescribed template’s geometry for data with known or emphasized structure.The design can constrain approximations to retain a chosen template shape.Source: C A new distance metric that preserves… · p. 25
The geometric-template source adds Gaussian noise around a fixed embedding, while similarity transformations allow translation, rotation, and uniform scaling but exclude reflection, shear, and nonlinear deformation.The noise makes the source full-dimensional, and its scale controls thickness around the template.Source: C A new distance metric that preserves… · p. 26
Training first fits the similarity-constrained velocity field with fixed noise scales, then jointly fine-tunes those scales and velocity parameters by likelihood; the distance is the fitted distributions’ Jeffreys divergence.The second stage adapts the template noiseless to the observations.Source: C A new distance metric that preserves… · p. 26

C.1 Demonstration of a new distance metric that preserves the geometry of a template

On noisy, differently oriented square data, the similarity-constrained model better recovers the ground truth and preserves square geometry, whereas an unconstrained nonlinear model distorts it.The comparison is illustrated in Figure S12.Source: §C.1 · p. 26

D.1.1 Flow matching training

Affine-flow regularization addresses the growth to O(d^2) learned matrix entries, while unconstrained flows use training/validation splits and early stopping instead.The affine candidates are selected using unpenalized validation Flow Matching loss.Source: §D.1.1 · p. 27

D.1.2 Velocity network architecture

Affine Jeffreys estimates use integrated Gaussian moments, while unconstrained estimates use endpoint samples and the continuous change-of-variables identity.Condition embeddings include periodic representations that keep orientations across the boundary nearby in feature space.Source: §D.1.2 · p. 28

D.1.3 Gaussian-process with kernel regression

The GKR estimator fits smoothly condition-dependent response distributions and uses separate inputs and kernels for Fisher, synthetic OU, orientation, and neural-data settings.Its documented default uses 200 inducing points and a one-third validation fraction.Source: §D.1.3 · p. 29

D.1.4 Wishart-process estimator

The Wishart process models condition-varying means and covariances; for PCA-70 Neuropixels, kernel selection was difficult, and 10-component responses gave more reasonable fits.Affine flow matching remains best among Gaussian methods in that comparison.Source: §D.1.4 · pp. 29–30

D.2.1 Categorical RDM dataset

This experiment evaluates categorical RDM estimation on five conditions with known response distributions. It uses repeated samples from a fixed synthetic population and compares estimates against analytically computed ground-truth distances.

D.2.1 Categorical RDM dataset

The synthetic population has five categories with three-dimensional responses, and its conditional means and diagonal variances are fixed across five repetitions.Each repetition differs only in its sampled observations.Source: §D.2.1 · p. 31
Category means are generated randomly and resampled until every pair is separated by at least 3/2, while conditional variances are specified separately.The supplied passage identifies 3/2 as half the diagonal of the unit cube.Source: §D.2.1 · p. 31
Performance is measured by mean relative absolute error over unique off-diagonal pairs, with accuracy curves by sample size and RDM estimates shown at 2,000 total samples.The comparison includes flow-matching and plug-in or covariance-based baselines, as well as TRE with 1, 8, or 16 bridges.Source: §D.2.1 · pp. 31–32

D.2.2 Fisher datasets

This appendix specifies synthetic Fisher-information and time-resolved distance benchmarks, documents neural recording datasets, and describes a geometry-template estimator alongside supplementary diagnostics.

D.2.2 Fisher datasets

Gaussian benchmarks target linear Fisher information, while the two-scale mixture changes higher-order structure without changing mean or covariance and targets full Fisher information.The sweeps vary sample size or response dimension and compare Unconstrained Flow, Affine Flow, GKR, and Wishart.Source: §D.2.2 · pp. 32–33

D.2.3 Time-resolved Jeffreys datasets

Time-resolved benchmarks include Gaussian OU trajectories and two-scale mixtures whose equal-probability branches preserve base moments but vary higher-order structure across conditions.Benchmark sweeps vary trial count or dimension and compare classical estimators with affine or unconstrained flow.Source: §D.2.3 · pp. 33–34

D.3.1 Mouse V1 static-grating dataset

Six mouse V1 sessions provide static-grating responses; five conditional-density estimators are compared by held-out log likelihood, not direct Fisher-information accuracy.Responses are split by orientation into training, validation, and test sets, then projected onto 200 principal components.Source: §D.3.1 · p. 34

D.3.2 Allen Brain Observatory two-photon calcium-imaging dataset with drifting gratings

Eleven Allen two-photon sessions span 40 direction–frequency conditions; observations are conditioned on continuous time, with frames from the same trial allowed across data splits.Each condition has 14–15 repeated trials, and the analysis retains 95 principal components.Source: §D.3.2 · pp. 34–35

D.3.3 Allen Visual Coding Neuropixels dataset

The Neuropixels analysis covers 32 sessions and 40 direction–frequency conditions, using 40 time bins and 70 principal components before random train/test splitting.Held-out likelihood is aggregated across conditions and sessions; the 4-Hz direction RDMs are evaluated at three representative times.Source: §D.3.3 · p. 35

D.4 Estimator and dataset for the geometry-template Jeffreys distance

The geometry-template benchmark compares similarity-constrained flows, which can only translate, rotate, and uniformly rescale a square, with unconstrained flows that may deform it; the constrained fit preserves square geometry.Both models start from the same noisy, side-length-one square source and are fit to noisy observations from two rotated square conditions.Source: §D.4 · p. 35, Figure S12 · p. 48

E Supplementary Figures

Supplementary Figures S1–S6 cover estimator error and ground-truth correlations, Stringer density fits and Fisher information, Wishart width selection, and Allen time-resolved density fits.The figures include density-fit comparisons across methods and validation-based kernel-width selection.Source: Figure S1 · p. 37, Figure S2 · p. 38, Figure S3 · p. 39
Supplementary Figures S7–S13 visualize frequency-specific and all-condition Neuropixels RDMs, conditional-density fits, held-out likelihood, two-photon estimates, and cross-time RDM correlations.The figures include session-averaged results and representative time points or conditions.Source: Figure S7 · p. 43, Figure S8 · p. 44, Figure S9 · p. 45