Source-linked AI summary
Learning Deep Representations with Probabilistic Knowledge Transfer
Nikolaos Passalis, Anastasios Tefas
TL;DR
Existing knowledge-transfer methods are mainly designed for classification, leaving representation-learning transfer across architectures, tasks, and modalities limited. The paper proposes Probabilistic KT, which matches feature-space probability distributions to preserve mutual information and geometry, and reports stronger performance across varied transfer setups.
Problem
Existing KT methods are tailored toward classification and cannot be used efficiently for other representation-learning tasks.
Method
Probabilistic KT transfers knowledge by matching the probability distribution of data in teacher and student feature spaces rather than their actual representations.
Results
PKT outperforms several other KT techniques across deep-neural-network, handcrafted-feature, and different-modality evaluation setups.
Takeaways & Limitations
PKT supports cross-modal transfer, transfer from handcrafted feature extractors into neural networks, and transfer across tasks and feature spaces of different dimensionality.
Takeaways & Limitations
Directly achieving the stated divergence objective can be infeasible when the student has significantly fewer parameters, and teacher and student require appropriately tuned kernel widths.
Abstract
from arXiv · showhide
Knowledge Transfer (KT) techniques tackle the problem of transferring the knowledge from a large and complex neural network into a smaller and faster one. However, existing KT methods are tailored towards classification tasks and they cannot be used efficiently for other representation learning tasks. In this paper a novel knowledge transfer technique, that is capable of training a student model that maintains the same amount of mutual information between the learned representation and a set of (possible unknown) labels as the teacher model, is proposed. Apart from outperforming existing KT techniques, the proposed method allows for overcoming several limitations of existing methods providing new insight into KT as well as novel KT applications, ranging from knowledge transfer from handcrafted feature extractors to {cross-modal} KT from the textual modality into the representation extracted from the visual modality of the data.
1 Introduction
Knowledge transfer trains smaller student networks from larger teacher models, but existing methods are largely restricted to classification and struggle with representation learning. The paper proposes Probabilistic KT, which matches feature-space probability distributions to preserve information and geometry across diverse transfer settings.
- Knowledge Transfer trains a smaller student model from a larger teacher model, typically by regressing the teacher’s output or a transformed output.
- Existing KT methods are tailored to classification outputs, limiting direct transfer across different architectures and representation-learning tasks.
- The paper investigates transferring feature-space geometry, handcrafted features, and knowledge across modalities and tasks beyond classification.
- Probabilistic KT models interactions among samples as an affinity probability distribution and trains the student to regress the teacher’s distribution.
- Matching teacher and student feature-space distributions is motivated by preserving quadratic mutual information and recreating local feature-space geometry.
- The proposed technique is evaluated across deep-network, handcrafted-feature, cross-modal, and object-detector transfer setups.
2 Related Work
Earlier KT methods mainly transfer soft labels, initialize student weights, or use intermediate-layer hints. The paper positions PKT as a probabilistic representation-learning method that avoids fixed output dimensions and domain-specific temperature tuning.
- Most prior KT methods use teacher-generated soft labels to train smaller student networks.
- Other approaches transfer knowledge through teacher-weight initialization or intermediate-layer hints, using projections when feature dimensions differ.
- PKT directly matches probability distributions between teacher and student feature spaces using a divergence metric for representation learning.
- PKT is designed to transfer across different dimensionalities without lossy projections and without requiring teacher-model details beyond the induced probability distribution.
3 Probabilistic Knowledge Transfer
Probabilistic Knowledge Transfer models relationships among samples in teacher and student feature spaces as probability distributions, then trains the student to match them. The method uses conditional probabilities and divergence minimization to preserve local geometry while connecting transfer to mutual information.
- The transfer set contains N objects used to transfer knowledge between the teacher and student models.
- Teacher and student outputs define continuous random variables X and Y representing their respective feature spaces.
- Pairwise joint density probabilities model feature-space geometry, and minimizing divergence between teacher distribution P and student distribution Q transfers that geometry.
- Matching teacher and student probability distributions preserves neighbors and relative distances, maintaining geometric relationships in the student’s lower-dimensional feature space.
- Because reproducing a significantly smaller teacher geometry can be infeasible, the method uses conditional probability distributions to provide a more flexible dimensionality-reduction-style transfer.
- Cosine similarity is used instead of Gaussian-kernel bandwidth tuning, and KL divergence emphasizes matching neighboring points during optimization.
- The student is optimized by gradient descent with Adam, using conditional probabilities estimated from small batches of 64–128 samples.
- The method connects probability-distribution matching with preserving the teacher’s quadratic mutual information between representations and possible unknown labels.
4 Experimental Evaluation
Experiments evaluate PKT across neural-network, handcrafted-feature, cross-modal, and object-detection settings. PKT improves retrieval and detection performance over the reported baselines and supports supervised augmentation.
- KT from Handcrafted Feature Extractors: 54.84% mAP is obtained with PKT from handcrafted CS-LBP features, exceeding the 52.31% hint-based result and 46.38% LBP baseline.The CS-LBP teacher representation has 480 dimensions, while the student learns a smaller representation.
- KT from Handcrafted Feature Extractors: PKT supports supervised augmentation by constructing a probability distribution that assigns p_i|j = 1 to same-class pairs and 0 otherwise.This allows transferred representations to be finetuned with supervised or other domain-specific information.
- Cross-modal KT: PKT transfers textual attributes into a visual student network and produces over 80% relative mAP improvement over hint-based transfer.The cross-modal setup is evaluated on the SUN Attribute dataset.
- KT from Object Detectors: 44.14% mAP is achieved for PKT object detection, outperforming the evaluated alternatives on PASCAL VOC.The comparison includes random initialization and initialization from matching Darknet layers trained on ImageNet.
5 Conclusions
The paper proposes Probabilistic KT, which matches feature-space probability distributions rather than representations. Across neural, handcrafted, and multimodal transfer settings, it outperforms several other KT techniques.
- PKT transfers knowledge by matching probability distributions of data in feature space instead of directly matching representations.
- Across deep-neural-network, handcrafted-feature, and different-modality setups, PKT outperforms several other KT techniques.