PaperlayerView exact revision on arXiv

Source-linked AI summary

Relational Knowledge Distillation Brings DNN Representations Close Enough to Humans to Be Aligned Without Supervision

Yuria Shimizu, Soh Takahashi, Takato Horii, Masafumi Oizumi

arXiv:2608.27877v1cs.CVq-bio.NC

TL;DR

Whether transferring human relational structure can align DNNs with humans at the individual-object level and generalize beyond training concepts remained unclear. Using RKD and unsupervised GWOT evaluation, the authors find improved coarse-category and individual-object matching on an independent test set of non-overlapping concepts.

  • Problem

    It remained unclear whether transferring human relational structure extends to individual-object alignment and an independent human embedding, a key issue for DNNs as models of vision.

    Source: §1 · pp. 2, 3, 5

  • Method

    The study fine-tunes pretrained DNNs with RKD and tests alignment using unsupervised GWOT on an independently derived human embedding with non-overlapping concepts.

    Source: §1 · pp. 5–6

  • Results

    On the independent, non-overlapping test set, RKD increased GWOT matching accuracy at coarse-category and individual-object levels, while human–DNN nearest-neighbor overlap remained largely unchanged.

    Source: §3.2.1 · p. 24, §4 · p. 26

  • Takeaways & Limitations

    The findings support the conclusion that relational transfer can bring DNN structure close enough for fine-grained human alignment without predefined object correspondences.

    Source: §1 · p. 5, §4 · p. 26

    Prior evaluations used supervised comparison with predefined object pairings, so they did not establish inferred individual-object correspondences or generalization beyond training concepts.

    Source: §1 · pp. 3, 5

Abstract

from arXiv · show

Linking the internal representations of deep neural networks (DNNs) to human mental representations is important for using DNNs as computational models of human vision. Existing DNN representations remain insufficiently similar to human mental representations, which are not directly observable and are therefore commonly measured through large-scale similarity judgments of object images. A natural approach to narrowing this gap is to directly transfer the relational structure of human representations into DNNs, and previous studies have reported improved human-DNN representational similarity. However, whether this improvement holds under stricter evaluation remains untested in two respects: fine-grained alignment at the individual-object level, and generalization to a human embedding derived from a dataset independent of the training data. Here, we employ an unsupervised comparison method, Gromov-Wasserstein optimal transport (GWOT), which estimates human-DNN correspondences from the internal distance structure alone and thereby tests fine-grained alignment. We further assess generalization on a curated test set of concepts non-overlapping with the training data. We show that fine-tuning pre-trained DNNs with Relational Knowledge Distillation (RKD), an established relational transfer method, brings DNNs close enough to humans to be aligned at the individual-object level on this test set. We also show that this improvement is driven by a more human-like global structure, as reflected in the ordering of distances among coarse categories, while the local human-DNN nearest-neighbor overlap rate remains largely unchanged. These findings indicate that relational transfer from humans brings the global structure of pre-trained DNNs close enough to the human structure to enable fine-grained human-DNN alignment without supervision.

1. Introduction

The study tests whether transferring human relational structure into DNNs enables unsupervised, individual-object alignment, including on an independent human embedding with non-overlapping concepts. It reports that RKD improves global structural correspondence, while local nearest-neighbor agreement remains largely unchanged.

Motivation

  • DNNs predict neural responses and recognition behavior, but this does not establish that they represent objects as humans do; classification accuracy and model scale do not predict agreement with human similarity judgments.Human mental representations are commonly measured through similarity judgments, which capture relational structure among objects.Source: §1 · pp. 2–3

Evaluation gap

  • Conventional supervised comparisons assume a stimulus pairing, so apparent similarity may persist even when alignment is only at the coarse-category level, leaving fine-grained generalization untested.Earlier evaluations, including those using separate human-similarity datasets, relied on supervised comparison.Source: §1 · pp. 3, 5

Unsupervised comparison

  • GWOT infers human–DNN object correspondences from internal distance structures, testing whether each human object’s most probable match is the identical DNN object.The same transport plan can also be scored at the coarser category level.Source: Figure 1 · p. 4, §1 · p. 5

Independent evaluation

  • Training used human similarity judgments on ImageNet images, while evaluation used a THINGS-derived embedding with overlapping concepts removed using a WordNet-based criterion.The training embedding came from Roads & Love (2021), and the evaluation embedding from Hebart et al. (2023).Source: §1 · p. 6

Main findings

  • RKD fine-tuning brought DNNs into individual-object correspondence with humans on an evaluation embedding curated to exclude training concepts.The study used relational transfer to fine-tune pre-trained DNNs and evaluated them against an independent human psychological embedding.Source: §1 · p. 6
  • The fine-grained gain was traced to more human-like global structure, reflected in coarse-category distance ordering, while local neighborhood agreement did not increase.The paper reports that local human–DNN nearest-neighbor overlap remained unchanged.Source: §1 · p. 6

2. Methods

The methods transfer human relational structure into pretrained DNNs with RKD trained on ImageNet, then evaluate alignment on non-overlapping THINGS concepts using unsupervised matching and local- and global-structure analyses.

2.1.1. Training dataset: ImageNet

  • The ImageNet teacher was a 125-dimensional Euclidean embedding fitted to human similarity judgments on 50,000 validation images, with dimensionality and initialization selected on held-out validation judgments.The training judgments comprised 0.50 million responses from 3,688 participants.Source: §2.1.1 · p. 8

2.1.2. Evaluation dataset: THINGS

  • The evaluation used the publicly released 66-dimensional THINGS SPoSE embedding and excluded WordNet-related concepts overlapping ImageNet, leaving 1,249 test concepts.The exclusion removed 605 concepts: 334 identical, 212 hypernym, and 59 hyponym concepts.Source: §2.1.2 · p. 9

2.2. Deep neural networks

  • The primary students were five CLIP ViT-B/16 variants pretrained on LAION-2B, LAION-400M, DataComp-L, DataComp-XL, or the OpenAI dataset, represented by 512-dimensional projection-head embeddings.Seven additional models spanning other architectures and pretraining schemes were also examined.Source: §2.2 · p. 10

2.3. Relational Knowledge Distillation

  • RKD transfers human pairwise-distance and triplet-angle relations into CLIP vision encoders by fine-tuning the DNN itself, without a coordinate mapping or additional task-specific loss (Park et al., 2019).The loss combines distance-wise and angle-wise terms, with λD = 1 and λA = 2; models were trained for 50 epochs on ImageNet images.Source: §2.3 · pp. 11, 13

2.4.1. Dissimilarity matrix construction; 2.4.2. Supervised comparison: Representational Similarity Analysis

  • Evaluation dissimilarities combined min-max-normalized cosine and Euclidean distances with a 2:1 weighting; supervised RSA compared their matrices using Spearman correlation under fixed object pairing.RSA used unique object pairs from the strict upper triangle of the matrices.Source: §2.4.1 · p. 13, §2.4.2 · pp. 13–14

2.4.3. Unsupervised comparison: Gromov–Wasserstein optimal transport

  • GWOT estimates human–DNN correspondences without a fixed stimulus pairing, using each space’s internal distance structure to compute an optimal transport plan.Matching accuracy was assessed at individual-object and coarse-category levels, reporting top-1 and top-5 rates against chance.Source: §2.4.3 · pp. 14–16

2.4.4. Local structure: k-nearest-neighbor overlap rate

  • Local structure was assessed by the mean k-nearest-neighbor overlap rate between human and DNN spaces on the 1,249-object test set, for k = 5 and k = 10.The reported mean was accompanied by a 95% confidence interval across query objects.Source: §2.4.4 · p. 16

3. Results

On a non-overlapping 1,249-object test set, RKD improved human–DNN similarity and enabled stronger fine-grained GWOT matching. The gain reflected a more human-like global structure, while local human–DNN neighborhood overlap remained largely unchanged.

3.1.1. Supervised comparison: RSA correlation increased after RKD

  • RSA Spearman correlation rose for all five models, from a pre-RKD range of ρ = 0.407–0.445 to a post-RKD range of ρ = 0.527–0.568.The authors note that overall RSA correlation alone cannot establish whether alignment is at the individual-object or only coarse-category level.Source: §3.1.1 · p. 18, Table 1 · p. 19

3.1.2. Unsupervised comparison: GWOT matching accuracy increased not only at the coarse-grained level but also at the fine-grained, individual-object level after RKD

  • After RKD, fine-grained GWOT top-1 matching accuracy rose from 0.560%–2.96% to 13.1%–19.5%, and coarse-grained accuracy rose from 34.8%–40.6% to 57.6%–61.2% across five variants.Fine-grained chance was 0.0801%; coarse-grained chance was 18.4%. Top-5 matching accuracies also showed comparable gains at both levels.Source: §3.1.2 · pp. 18–19, Table 1 · p. 19
  • The GWOT gains generalized beyond CLIP ViT-B/16 and CLIP-style image–text pre-training: RSA correlation and GWOT matching accuracy both rose after RKD across models with different architectures and pre-training schemes.Coarse-level GWOT gains also held when objects were grouped into 21 clusters derived by k-means from the human embedding, rather than human-annotated categories.Source: §3.1.2 · p. 21

3.1.3. Control experiment: a within-dataset split of THINGS showed how strict our generalization test was

  • In the within-THINGS control, RSA correlation rose from 0.448–0.464 to 0.803–0.825, individual-object top-1 accuracy from 0.00%–1.33% to 58.0%–65.0%, and coarse-category top-1 accuracy from 16.0%–24.7% to 77.7%–84.7%.Because this control used 300 test objects rather than 1,249, the paper cautions against direct numerical comparison; all measures were substantially higher in the control, indicating a stricter primary generalization test.Source: §3.1.3 · pp. 21–22

3.2.1. Local structure: human–DNN nearest-neighbor overlap rate remained largely unchanged after RKD

  • Human–DNN k-nearest-neighbor overlap remained largely unchanged after RKD: mean changes ranged from −4.2 to +0.8 percentage points, despite pre-RKD k = 5 overlap of 25.9%–28.5% against 0.4% chance.RKD replaced the majority of each object’s DNN nearest neighbors, but the new neighbors agreed with human neighborhoods just as well as those they replaced; the gain in GWOT matching therefore was not attributable to more human-like local neighborhoods.Source: §3.2.1 · pp. 22–24

3.2.2. Global structure: the coarse-category arrangement became more human-like after RKD

  • After RKD, human–DNN Spearman correlation between inter-coarse-category dissimilarity matrices rose from ρ = 0.572–0.751 to ρ = 0.838–0.879 across five variants.For CLIP ViT-B/16 (LAION-2B), it rose from ρ = 0.572 to ρ = 0.878; category arrangements also became more human-like, and the pattern held for data-driven clusters as well as human-annotated categories.Source: §3.2.2 · pp. 24–26

4. Discussion

RKD made human–DNN representations matchable at the individual-object level under unsupervised comparison on concepts held out from transfer. The improvement reflected more human-like global category structure, while local nearest-neighbor overlap remained largely unchanged.

4. Discussion

  • 13.1%–19.5% after RKD versus 0.560%–2.96% before, GWOT individual-object matching accuracy increased on the independent test set.The transport plan showed a visible identity diagonal only after RKD.Source: §4 · p. 26, §4.1 · p. 27

4.1. Unsupervised comparison is necessary for evaluating relational transfer at the fine-grained, individual-object level

  • RSA rose from ρ = 0.407–0.445 to ρ = 0.527–0.568 after RKD, but alone could not establish fine-grained alignment.For CLIP ResNet-50, RSA changed only from ρ = 0.500 to ρ = 0.501, while individual-object matching rose from 0.400% to 10.3%.Source: §4.1 · p. 27

4.2. Relational transfer conveys a general organization of object representations, not adjustments specific to the trained concepts

  • Alignment generalized to concepts without WordNet hierarchical relations to training concepts and to an independently obtained embedding from a different judgment task and image dataset.A within-dataset THINGS split without these demands yielded substantially higher measures, yet individual objects remained matchable under the primary evaluation.Source: §4.2 · pp. 27–28

4.3. Relational transfer from humans supplies the human-like global structure that does not emerge spontaneously from pre-training

  • Coarse-category distance-order correlation increased from ρ = 0.572–0.751 to ρ = 0.838–0.879 after transfer, while local nearest-neighbor overlap remained largely unchanged.This dissociation fits the Aristotelian view: pre-transfer local overlap was 25.9%–28.5% against 0.4% chance, whereas coarse-category GWOT accuracy rose from 34.8%–40.6% to 57.6%–61.2%.Source: §4 · p. 26, §4.3 · p. 28

4.4. Future work: toward transferring neural and person-specific representations

  • Future work could transfer neural relational structure with behavioral data using the same RKD loss and architecture, and adapt population embeddings to individuals, though same-person data collection is a bottleneck.The transferred behavioral embedding pooled judgments across thousands of participants; neural measurements may complement behavioral judgments for individual-level transfer.Source: §4.4 · p. 29

CRediT authorship contribution statement

Yuria Shimizu contributed across the research and writing, while Soh Takahashi and Takato Horii contributed to methodology and manuscript review. Masafumi Oizumi led supervision, resources, project administration, and funding acquisition.

S2. RKD applied to DNNs with different architectures and pre-training schemes · S5. Few-shot classification performance before and after RKD · S1. Coarse-category-level analyses using data-driven clusters

With 21 data-driven clusters, RKD improved coarse-category matching and made CLIP’s inter-cluster arrangement more like the human embedding, reproducing patterns found with human-annotated categories.

S1. Coarse-category-level analyses using data-driven clusters

  • The analysis used 21 k-means clusters from the 66-dimensional human embedding of all 1,249 test objects, matching the annotated grouping’s granularity.The data-driven clusters overlapped only partially with the human-annotated categories.Source: S1. Coarse-category-level analyses… · p. 41
  • Figure S1 reports data-driven cluster-level top-1 GWOT matching accuracy before and after RKD for five CLIP ViT-B/16 variants, with a dotted 5.8% chance reference.Source: Figure S1 · p. 42
  • ρ = 0.846–0.874 post-RKD inter-cluster Spearman correlation across five variants, up from ρ = 0.636–0.771, alongside projections resembling humans rather than pre-RKD CLIP.For CLIP ViT-B/16 (LAION-2B), correlation rose from ρ = 0.636 to ρ = 0.857; annotated-category correlations were ρ = 0.572–0.751 before and ρ = 0.838–0.879 after RKD.Source: S1. Coarse-category-level analyses… · pp. 41, 43, Figure S1 · p. 42

S2. RKD applied to DNNs with different architectures and pre-training schemes

RKD improved human–DNN alignment across architectures and pre-training schemes, but substantial gains depended on the initial representation being aligned with humans to some extent. RSA and GWOT did not always move together, with GWOT detecting gains that RSA could miss.

Experimental scope and procedure

  • The evaluation applied the same RKD procedure and RSA/GWOT tests to seven additional models spanning ResNet and ViT architectures and four pre-training schemes.Models were evaluated before and after RKD on the 1,249-object THINGS test set; distillation used the same human teacher embedding, ImageNet training images, objective, and 50-epoch schedule.Source: S2. RKD applied to DNNs with different… · pp. 43–44

Results

  • For CLIP ResNet-50, RSA barely changed (ρ = 0.500 to ρ = 0.501), while GWOT accuracy rose substantially at both granularity levels.The authors note that supervised comparison alone can miss a representational reorganization detected by unsupervised comparison.Source: S2. RKD applied to DNNs with different… · p. 44
  • MAE ViT-B/16 had the weakest human alignment before RKD and the smallest gain in GWOT matching accuracy, though both RSA and GWOT measures rose.This pattern supports the conclusion that a representation needs some initial alignment with humans for RKD to yield a substantial gain.Source: S2. RKD applied to DNNs with different… · pp. 44–45
  • For the randomly initialized ViT-B/16, individual-object top-1 accuracy rose from 0.00% to 0.160% (chance: 0.0801%), while coarse-category top-1 reached 20.7% (chance: 18.4%).Alignment improved, but remained close to chance; the authors conclude that initial human alignment is needed for RKD to produce a substantial gain.Source: S2. RKD applied to DNNs with different… · pp. 44–45
  • Both RSA correlation and GWOT matching accuracy rose after RKD for all six pretrained models, with most post-RKD values comparable to the five CLIP ViT-B/16 variants.Gain sizes varied across models; Table S1 reports results on the 1,249-object THINGS test set.Source: S2. RKD applied to DNNs with different… · p. 44, Table S1 · p. 45

S3. Control experiment: a within-dataset split of THINGS

The within-THINGS control removed the primary evaluation’s cross-task, cross-dataset human-embedding and non-overlapping-concept demands. Its higher matching results show that the primary evaluation imposes a stricter generalization test.

S3. Control experiment: a within-dataset split of THINGS

  • Unlike the primary evaluation, the control used a conventional within-dataset split, removing both cross-task, cross-dataset human-embedding differences and the requirement that test concepts not overlap RKD training concepts.The primary evaluation used embeddings from different judgment tasks, image datasets, and embedding models.Source: S3. Control experiment: a… · pp. 45–46
  • After RKD, RSA correlation was ρ = 0.803–0.825, GWOT fine-grained top-1 matching accuracy 58.0%–65.0%, and coarse-grained top-1 matching accuracy 77.7%–84.7%.Before RKD, the corresponding ranges were ρ = 0.448–0.464, 0.00%–1.33%, and 16.0%–24.7%; all three measures rose for each of five CLIP ViT-B/16 variants.Source: S3. Control experiment: a… · p. 46
  • The control split the 1,854 THINGS objects alphabetically into 1,554 training and 300 test objects, then applied the same RKD objective and RSA and GWOT procedures.The same 21-category assignment, defined on all objects, was applied to the test set.Source: S3. Control experiment: a… · p. 46
  • GWOT fine-grained top-1 matching accuracy was 58.0%–65.0% under the within-dataset split versus 13.1%–19.5% in the primary evaluation, whose test sets differ.The authors therefore conclude that the primary evaluation is a far stricter generalization test.Source: S3. Control experiment: a… · p. 46

S4. Reorganization of the DNN’s own local neighborhoods after RKD

RKD substantially reorganized CLIP’s own local neighborhoods: across five variants, fewer than half of the nearest neighbors were retained on average. This contrasts with the largely unchanged human–DNN nearest-neighbor overlap rate.

  • For each of 1,249 THINGS test objects, the analysis measured how many pre-RKD CLIP neighbors remained among its post-RKD neighbors, then averaged across objects for each variant.The overlap was computed separately at each k, with means and 95% confidence intervals reported.Source: S4. Reorganization of the DNN’s own… · p. 47
  • Across five CLIP ViT-B/16 variants, neighbor retention was 33.8%–44.6% at k = 5 and 35.0%–45.5% at k = 10 after RKD.These are mean proportions retained across the test objects; the reported Table S3 values include 95% confidence intervals.Source: S4. Reorganization of the DNN’s own… · p. 47, Table S3 · p. 48
  • RKD substantially reorganized the DNN’s own local neighborhoods even though the human–DNN nearest-neighbor overlap rate remained largely unchanged.The comparison here is between pre-RKD and post-RKD CLIP neighbors, not between human and DNN neighbors.Source: S4. Reorganization of the DNN’s own… · pp. 47–48

S5. Few-shot classification performance before and after RKD · Supplementary References

RKD modestly reduced few-shot accuracy for most evaluated CLIP variants, while the OpenAI-pretrained variant was largely spared. The relational-only objective had no task-specific loss or preservation term, which could allow it to overwrite useful pre-trained structure.

S5. Few-shot classification performance before and after RKD

  • RKD optimized human relational agreement alone, without a task-specific loss or regularization term, and therefore did not anchor the model to its pre-trained structure.The authors argue that a preservation term would be warranted if maintaining downstream performance were an additional goal; prior work reported similar degradation without one (Muttenthaler, Linhardt, et al., 2023; Muttenthaler et al., 2025).Source: S5. Few-shot classification… · pp. 48, 50
  • The few-shot evaluation used prototype matching without fitting task parameters, with five randomly selected support examples per class and five paired draws before and after RKD.Prototypes averaged the support-example features; remaining images were assigned to the closest prototype by the article’s dissimilarity measure, following prototype matching (Snell et al., 2017).Source: S5. Few-shot classification… · p. 48
  • The four datasets spanned the alignment domain (THINGS), RKD training images (ImageNet), and two out-of-domain datasets (CIFAR-100 and CIFAR-20).THINGS has 1,854 classes, ImageNet 1,000, CIFAR-100 100, and CIFAR-20 20.Source: S5. Few-shot classification… · pp. 48–49
  • RKD modestly reduced few-shot accuracy for four variants: drops were largest on CIFAR-100 (8.6–12.3 percentage points) and ImageNet (8.3–9.0), and smallest on THINGS (2.7–4.5).Table S4 reports accuracy as mean ± SD across five paired draws of five support examples per class.Source: Table S4 · p. 49, S5. Few-shot classification… · p. 49
  • Those four variants retained high THINGS accuracy, declining from 85.6%–90.1% before RKD to 81.3%–87.4% after.The authors note that the ImageNet reduction was notable because RKD used those same images.Source: S5. Few-shot classification… · p. 49
  • The OpenAI-pretrained CLIP ViT-B/16 variant was an exception: accuracy was essentially unchanged on THINGS and CIFAR-100, rose on CIFAR-20, and fell modestly on ImageNet.The table covers five CLIP ViT-B/16 variants across THINGS, ImageNet, CIFAR-100, and CIFAR-20.Source: S5. Few-shot classification… · p. 49, Table S4 · p. 49
  • The authors conclude that human relational structure does not always benefit few-shot accuracy, a standard benchmark for pre-trained CLIP models (Cherti et al., 2023).They note that aligning representations to human judgments without an explicit preservation term had also degraded few-shot performance (Muttenthaler, Linhardt, et al., 2023).Source: S5. Few-shot classification… · pp. 49–50

Related papers

Loading 2608.27877v1…