Relational Knowledge Distillation Brings DNN Representations Close Enough to Humans to Be Aligned Without Supervision

Yuria Shimizu; Soh Takahashi; Takato Horii; Masafumi Oizumi

Source-linked AI summary

Relational Knowledge Distillation Brings DNN Representations Close Enough to Humans to Be Aligned Without Supervision

Yuria Shimizu, Soh Takahashi, Takato Horii, Masafumi Oizumi

arXiv:2608.27877v1Aug 28, 2026cs.CVq-bio.NC

TL;DR

DNN representations remain insufficiently similar to human mental representations, and prior evaluations did not establish fine-grained alignment or generalization to independent human embeddings. The paper fine-tunes pre-trained DNNs with RKD and evaluates them using GWOT on non-overlapping concepts. RKD enables individual-object alignment without supervision, with the gain linked to more human-like global structure rather than changed nearest-neighbor overlap.

Problem
Existing DNN representations remain insufficiently similar to human mental representations, while prior supervised evaluations could not establish individual-object alignment or independent generalization.
Method
The study transfers human relational structure by fine-tuning pre-trained DNNs with RKD, then uses GWOT to infer correspondences from internal distances on an independent, non-overlapping test embedding.
Results
RKD increased human–DNN similarity and enabled unsupervised matching at both coarse-category and individual-object levels on the independent test set.
Takeaways & Limitations
Relational transfer can bring pre-trained DNN global structure close enough to human structure for fine-grained human–DNN alignment without supervision.
Takeaways & Limitations
RKD optimizes agreement with human relational structure without preserving the pre-trained representation, so downstream few-shot performance may degrade without an explicit preservation term.

Abstract

from arXiv · show

Linking the internal representations of deep neural networks (DNNs) to human mental representations is important for using DNNs as computational models of human vision. Existing DNN representations remain insufficiently similar to human mental representations, which are not directly observable and are therefore commonly measured through large-scale similarity judgments of object images. A natural approach to narrowing this gap is to directly transfer the relational structure of human representations into DNNs, and previous studies have reported improved human-DNN representational similarity. However, whether this improvement holds under stricter evaluation remains untested in two respects: fine-grained alignment at the individual-object level, and generalization to a human embedding derived from a dataset independent of the training data. Here, we employ an unsupervised comparison method, Gromov-Wasserstein optimal transport (GWOT), which estimates human-DNN correspondences from the internal distance structure alone and thereby tests fine-grained alignment. We further assess generalization on a curated test set of concepts non-overlapping with the training data. We show that fine-tuning pre-trained DNNs with Relational Knowledge Distillation (RKD), an established relational transfer method, brings DNNs close enough to humans to be aligned at the individual-object level on this test set. We also show that this improvement is driven by a more human-like global structure, as reflected in the ordering of distances among coarse categories, while the local human-DNN nearest-neighbor overlap rate remains largely unchanged. These findings indicate that relational transfer from humans brings the global structure of pre-trained DNNs close enough to the human structure to enable fine-grained human-DNN alignment without supervision.

1. Introduction

DNNs do not yet reliably match human object representations at the individual-object level or generalize across independently derived human embeddings. This paper addresses both gaps by transferring human relational structure and evaluating alignment without predefined object pairings.

DNN representations capture human object-similarity structure only to a limited extent, and standard accuracy or scale does not eliminate this limitation.
Prior relational adaptations improved human–DNN similarity, but conventional supervised comparison could not establish fine-grained individual-object correspondence.Supervised comparison supplies the object pairing in advance, so high similarity can persist even when alignment is only at the coarse-category level.
GWOT infers human–DNN correspondences from internal distance structures alone, then evaluates the resulting transport plan at individual-object and coarse-category levels.The method estimates an optimal transport plan whose entries represent correspondence probabilities between human and DNN objects.
The study trains RKD on ImageNet human similarity judgments and evaluates pre-trained and fine-tuned DNNs on a separate THINGS embedding after removing overlapping concepts.The training embedding uses 50,000 ImageNet stimuli, while the independent evaluation embedding contains 1,854 THINGS concepts.
RKD increased supervised similarity and unsupervised matching at both coarse-category and individual-object levels on the independent test set.The authors report that fine-grained alignment emerged under the stricter GWOT evaluation.
The fine-grained gain was associated with a more human-like global structure, whereas human–DNN nearest-neighbor overlap remained largely unchanged.RKD replaced most DNN nearest-neighbor sets, yet the newly introduced neighbors agreed with human neighborhoods about as well as those they replaced.

3. Results

On a non-overlapping 1,249-object test set, RKD improved human–DNN correspondence under both supervised RSA and unsupervised GWOT, including individual-object matching. The gain was associated with a more human-like global category structure, while local nearest-neighbor overlap remained largely unchanged.

3.1.1. Supervised comparison: RSA correlation increased for all five models after RKD, from ρ = 0.407–0.445 to ρ = 0.527–0.568.RSA alone could not determine whether the improvement occurred at the individual-object or coarse-category level.
3.1.2. Unsupervised comparison: Fine-grained GWOT top-1 matching rose from 0.560%–2.96% to 13.1%–19.5% across five variants, versus 0.0801% chance.This increase was more than two orders of magnitude above chance, whereas before RKD almost no individual object could be recovered.
3.1.2. Unsupervised comparison: Coarse-grained GWOT top-1 matching rose from 34.8%–40.6% to 57.6%–61.2%, with comparable gains for top-5 matching.The coarse-grained chance level was 18.4%.
3.1.2. Unsupervised comparison: After RKD, GWOT transport plans showed clearer coarse-category blocks and a stronger diagonal corresponding to individual-object matches.For example, “rhinoceros” received mostly unrelated inanimate-object targets before RKD, but animal targets including “rhinoceros” itself afterward.
3.2.1. Local structure: Human–DNN k-nearest-neighbor overlap changed by only −4.2 to +0.8 percentage points across variants and k values, despite RKD replacing most DNN neighbor sets.The pre-RKD k = 5 overlap was already 25.9%–28.5%, compared with 0.4% chance.
3.2.2. Global structure: Inter-coarse-category Spearman correlation increased from ρ = 0.572–0.751 before RKD to ρ = 0.838–0.879 afterward, indicating a more human-like global arrangement.Together with the unchanged nearest-neighbor overlap, these results indicate that the individual-object matching gain reflects global rather than local reorganization.

4. Discussion

RKD improved fine-grained human–DNN alignment under a strict, independent evaluation, with gains associated with a more human-like global structure rather than local-neighborhood changes. The discussion also identifies extensions to neural and individual-level human representations.

RKD’s fine-grained matching gain was associated with a more human-like arrangement among coarse categories, whereas nearest-neighbor overlap remained largely unchanged.The unchanged local overlap indicates that the reported gain was not attributable to altered local structure, while coarse-category arrangement became more human-like.
GWOT matching accuracy increased at the individual-object level after RKD, while RSA alone could not distinguish fine-grained from coarse alignment.Across models, individual-object accuracy rose from 0.560%–2.96% to 13.1%–19.5%; CLIP ResNet-50 illustrates the distinction, with RSA nearly unchanged but matching accuracy increasing.
The strict evaluation tested generalization to non-overlapping concepts and an independently obtained human embedding, yet individual objects remained matchable without supervision.The evaluation embedding came from a different judgment task, image dataset, and embedding model, while test concepts excluded hierarchical relations to training concepts.
Before RKD, human–DNN local neighborhood agreement was already above chance, while coarse-category arrangement was only partly human-like.For k = 5, local overlap was 25.9%–28.5% against 0.4% chance; coarse-category arrangement was ρ = 0.572–0.751.
The pipeline can transfer behavioral, neural, or combined relational data without changing the RKD loss or DNN architecture.RKD uses pairwise distances and triplet-wise angles, supporting the proposed extension from psychological embeddings to neural measurements.
Transferring an individual person’s embedding would make a DNN a person-specific proxy, but collecting the required complete behavioral judgments would be impractical.The study’s embedding pools judgments across thousands of participants; an individual embedding would require all judgments from one person.

CRediT authorship contribution statement

The contribution statement attributes the paper’s work across writing, visualization, validation, software, methodology, investigation, analysis, data curation, conceptualization, supervision, resources, project administration, and funding acquisition.

Yuria Shimizu contributed across writing, visualization, validation, software, methodology, investigation, formal analysis, data curation, and conceptualization.
Soh Takahashi and Takato Horii contributed to writing and methodology.Masafumi Oizumi contributed to writing, supervision, resources, project administration, methodology, funding acquisition, and conceptualization.

Supplementary Information for

The supplementary information accompanies the paper “Relational Knowledge Distillation Brings DNN Representations Close Enough to Humans to Be Aligned Without Supervision.”

The supplementary information is associated with the paper on aligning DNN representations with humans through relational knowledge distillation.The listed authors are Yuria Shimizu, Soh Takahashi, Takato Horii, and Masafumi Oizumi.

S1. Coarse-category-level analyses using data-driven clusters

Data-driven clusters reproduced the main coarse-category findings: RKD improved matching and made inter-cluster organization more human-like across all five model variants. These results did not depend on human-annotated THINGS categories.

S1. Coarse-category-level analyses using data-driven clusters: The analysis tested whether coarse-category results depended on human-annotated THINGS categories by repeating them with clusters derived from the human embedding.
S1. Coarse-category-level analyses using data-driven clusters: Twenty-one data-driven clusters were obtained by k-means clustering the 66-dimensional human embedding of 1,249 test objects.The cluster count matched the 20 THINGS coarse categories plus others, preserving the same granularity as the annotated grouping.
S1. Coarse-category-level analyses using data-driven clusters: GWOT transport plans were rescored using the data-driven clusters to repeat the coarse-category matching analysis.
S1. Coarse-category-level analyses using data-driven clusters: 58.7%–63.6% post-RKD data-driven cluster-level top-1 matching accuracy exceeded the 13.5%–30.7% pre-RKD range and 5.8% chance.The post-RKD range was comparable to the 57.6%–61.2% range obtained with human-annotated categories.
S1. Coarse-category-level analyses using data-driven clusters: The supplementary analyses repeated three-dimensional projections and Spearman correlations between human and CLIP 21 × 21 inter-cluster dissimilarity matrices.
S1. Coarse-category-level analyses using data-driven clusters: Post-RKD clusters were arranged more like the human embedding, unlike pre-RKD clusters, and inter-cluster correlations increased for all five variants.Across variants, correlations rose from ρ = 0.636–0.771 before RKD to ρ = 0.846–0.874 after RKD; the example model increased from ρ = 0.636 to ρ = 0.857.

S2. RKD applied to DNNs with different architectures and pre-training schemes

RKD was evaluated across seven additional DNNs spanning two architectures and four pre-training schemes. Alignment improved broadly after RKD, but gains depended on the starting representation and remained near chance for a randomly initialized model.

Model diversity: The seven additional models varied across convolutional ResNet and Vision Transformer architectures and image–text, supervised, self-supervised, and no pre-training schemes.Representations were extracted from each model’s final layer using architecture-appropriate features.
Alignment outcomes: RKD improved alignment for every model except the randomly initialized ViT-B/16, with gains varying widely across models.For six pre-trained models, both RSA correlation and GWOT matching accuracy increased after RKD.
Alignment outcomes: For CLIP ResNet-50, RSA correlation barely changed from ρ = 0.500 to ρ = 0.501, while GWOT matching accuracy rose substantially.This divergence shows that supervised comparison can miss reorganizations detected by unsupervised comparison.
Alignment outcomes: The randomly initialized ViT-B/16 remained close to chance after RKD: individual-object top-1 accuracy rose from 0.00% to 0.160% versus 0.0801% chance.Coarse-category top-1 accuracy reached 20.7% versus an 18.4% chance level.
Interpretation: RKD produced substantial gains only when the pre-trained representation already had some alignment with humans.The self-supervised MAE ViT-B/16 showed the smallest gain among the six pre-trained models.

S3. Control experiment: a within-dataset split of THINGS

A within-dataset THINGS split relaxed the primary evaluation’s dataset and concept-separation demands. Alignment measures were substantially higher in this control, indicating that the primary evaluation imposed a stricter generalization test.

Evaluation design: The primary evaluation required generalization across different image datasets, human judgment tasks, and embedding models.Its test concepts were also curated not to overlap with RKD training concepts.
Evaluation design: The control removed both demands by using a conventional THINGS object split without dataset or concept separation.The first 1,554 alphabetically sorted objects were used for training and the remaining 300 for testing.
Results: Every alignment measure was substantially higher under the within-dataset split than under the primary evaluation, although the test sets differed.The authors therefore do not compare the values directly as equivalent measurements.
Results: GWOT fine-grained top-1 matching accuracy was 58.0%–65.0% within-dataset versus 13.1%–19.5% in the primary evaluation.The paper identifies this measure as the primary measure of interest.

S4. Reorganization of the DNN’s own local neighborhoods after RKD

RKD substantially reorganized the DNNs’ own local neighborhoods, even though human–DNN nearest-neighbor overlap remained largely unchanged. The analysis compared pre-RKD and post-RKD CLIP neighborhoods independently of the human reference.

Analysis: The analysis compared each object’s pre-RKD and post-RKD CLIP nearest neighbors, independently of the human embedding.For each of 1,249 THINGS test objects, the retained-neighbor proportion was averaged across five CLIP ViT-B/16 variants.
Results: At k = 5, only 33.8%–44.6% of nearest neighbors were retained after RKD across the five variants.Values were reported as means with 95% confidence intervals in Table S3.
Results: At k = 10, only 35.0%–45.5% of nearest neighbors were retained after RKD.Thus, RKD replaced more than half of each object’s nearest neighbors on average.
Interpretation: This substantial reorganization of DNN neighborhoods occurred while human–DNN nearest-neighbor overlap remained largely unchanged.The comparison therefore distinguishes changes within the DNN representation from its local overlap with human neighborhoods.

S5. Few-shot classification performance before and after RKD

RKD reduced few-shot classification accuracy across most models and datasets, though the reductions were modest and one OpenAI-pre-trained CLIP variant was largely unaffected. This reflects a trade-off under an objective optimized solely for human relational structure.

Evaluation: Few-shot classification was measured before and after RKD without task-specific loss or regularization, quantifying the consequence of optimizing the relational objective alone.Class prototypes used five randomly selected examples per class, with paired selections before and after RKD.
Results: RKD reduced few-shot accuracy across models and datasets, but the reductions were modest overall.The evaluation covered THINGS, ImageNet, CIFAR-100, and CIFAR-20.
Results: For four variants, reductions were 8.6–12.3 percentage points on CIFAR-100 and 8.3–9.0 points on ImageNet, versus 2.7–4.5 points on THINGS.Those variants still reached 85.6%–90.1% accuracy on THINGS before RKD.
Interpretation: Because RKD lacked a term preserving the pre-trained structure, it could overwrite task-relevant representations without penalty.The authors state that an explicit preservation term would be warranted if maintaining downstream performance were an additional goal.