PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Kernels by Alignment for Multiclass Bayes Classification

Hollan Haule, Javier Escudero

arXiv:2609.06474v2cs.LG

TL;DR

Kernel methods often require the kernel to be selected and tuned, and extending kernel-based approaches to multiclass classification remains a challenge. The paper interprets CLaI as alignment-based kernel learning and kernel Bayes inference, then extends it with a learned Mahalanobis distance for multiclass tasks. The Mahalanobis formulation improves multiclass performance and convergence in the reported experiments, with lower calibration error than cosine similarity in sepsis prediction.

  • Problem

    Kernel methods commonly require kernel selection and tuning, while extending kernel-based methods to multiclass problems remains an active research interest.

    Source: §I · p. 1

  • Method

    CLaI learns an embedding aligned to a label-derived target kernel, then predicts unseen labels by similarity-weighted reference labels, with a learned Mahalanobis distance used in its multiclass extension.

    Source: §V · pp. 7–8

  • Results

    The learned Mahalanobis formulation improved multiclass performance and convergence in the experiments, and produced lower calibration error than cosine similarity in sepsis prediction.

    Source: §V · p. 8

  • Takeaways & Limitations

    CLaI links learned representations and kernel alignment with kernel Bayes classification, offering an alternative to choosing a similarity function independently of the task.

    Source: §V · pp. 7–8

    The multiclass extension has time complexity O(T·C·M^2), with the per-class embedding loop as its main additional cost.

    Source: §VI · p. 8

Abstract

from arXiv · show

Kernel methods separate data representation from decision-making, but typically require the kernel to be chosen in advance. We show that this kernel can instead be learned by alignment, and develop the resulting framework through the recently introduced Collaborative Learning and Inference (CLaI). We show that Collaborative Learning can be viewed as a kernel alignment process, in which an embedding is trained so that its induced similarity matches a label-derived target kernel. We also prove that Collaborative Inference is equivalent to kernel Bayes classification with Parzen-window density estimation. Motivated by these perspectives, we generalise CLaI by replacing cosine similarity with a learned Mahalanobis distance and extend it to multiclass classification. On CIFAR-10, PathMNIST, and SleepEDF, the Mahalanobis formulation improves accuracy, converges faster, and yields lower calibration error than the cosine-based variant. Auxiliary experiments further support these connections, showing that CLaI produces latent signals of the same form as a Gaussian process, while achieving competitive calibration on sepsis prediction. Together, these results establish a principled learned-kernel framework that unifies representation learning, kernel alignment, and Bayesian classification, and extends naturally to the multiclass setting.

I. INTRODUCTION

Kernel methods can make model selection costly because the kernel must be chosen and tuned, while extending kernel-based approaches to multiclass classification remains an active challenge. This introduction frames CLaI as a learned-kernel approach linking representation learning, kernel alignment, and Bayes classification, and outlines its Mahalanobis-distance and multiclass extensions.

I. INTRODUCTION

  • Kernel selection and tuning can demand substantial user intervention and make model selection computationally expensive and complex.SVMs also require cross-validation to tune kernel parameters.Source: §I · p. 1
  • CLaI is presented as a kernel framework in which learning aligns an embedding’s induced similarity with a label-derived target, while inference is equivalent to kernel Bayes classification with Parzen-window density estimation.The framework is positioned as an alternative to selecting a kernel a priori.Source: §I · p. 1
  • The paper extends CLaI to multiclass classification by conditioning φ on a target class under a one-versus-rest strategy.It also introduces a learned Mahalanobis-distance base kernel as the similarity function.Source: §I · p. 1
  • The introduction identifies theoretical connections to Bayes classification and kernel alignment, generalises the distance function using Mahalanobis distance, and reports multiclass validation on CIFAR-10, PathMNIST, and SleepEDF.These contributions connect learned representations with kernel-based classification.Source: §I · pp. 1–2

A. Kernel Methods fundamentals

  • Many supervised kernel methods use regularised empirical risk minimisation in an RKHS, where the Representer theorem expresses the minimiser as a finite combination of kernel functions centred at training points.The formulation separates the kernel-induced regularisation term from the task loss.Source: A. Kernel Methods fundamentals · p. 2

B. Deep Kernel Learning (DKL)

Deep kernel learning embeds learned representations within kernel methods, while kernel alignment compares a kernel matrix with a target similarity matrix. The approach section identifies connections among CLaI, kernel methods, and the Bayes classifier as part of its contributions.

B. Deep Kernel Learning (DKL)

  • Deep kernel learning embeds a deep-learning module within a base kernel to learn a mapping and kernel hyperparameters from data.The cited work trains these jointly via marginal likelihood and combines learned representations with an infinite-dimensional function space.Source: B. Deep Kernel Learning (DKL) · p. 2

C. Kernel Alignment

  • Kernel alignment measures similarity between kernel matrices and can evaluate kernel models for selection.For binary classification, the target matrix is T = yy^T; maximizing alignment is equivalent to minimizing the distance between kernel K and target T.Source: C. Kernel Alignment · p. 2

D. Bayes Classifier

  • The Bayes classifier minimizes expected misclassification error when class-conditional distributions are known or can be accurately estimated.Its binary-class rule uses class-conditional densities and corresponding class priors.Source: D. Bayes Classifier · p. 2

III. APPROACH AND THEORETICAL INSIGHTS

  • The approach identifies connections between CLaI, kernel methods, and the Bayes classifier, alongside a Mahalanobis-distance kernel and a multiclass extension.Source: §III · p. 2

A. CLaI: Collaborative Learning and Inference

CLaI learns an embedding in which same-class samples are close and different-class samples are separated, then predicts from proximity to a reference set. Its learning objective aligns cross-batch similarities with label-derived targets.

A. CLaI: Collaborative Learning and Inference

  • CLaI has two stages: Collaborative Learning trains an embedding, and Collaborative Inference assigns labels using proximity to reference samples.The formulation generalizes the earlier binary-classification approach, originally designed for time series, to other data types.Source: A. CLaI: Collaborative Learning and… · p. 2

1) Collaborative Learning:

  • The embedding model maps batches to latent representations, and its pairwise similarity function compares anchor and reference batches.The earlier implementation used cosine similarity.Source: 1) Collaborative Learning · p. 3
  • Training compares each anchor batch with distinct reference batches and accumulates their losses before updating model parameters.Duplicate reversed pairs and self-comparisons are omitted.Source: 1) Collaborative Learning · p. 3
  • Collaborative Learning minimizes kernel-alignment distance between learned cross-batch similarities and a label-induced target matrix.Same-class samples are encouraged to have high similarity and different-class samples low similarity.Source: 1) Collaborative Learning · p. 3

2) Collaborative Inference:

Collaborative Inference uses a trained embedding and labeled reference set to predict labels for unseen samples. It expects learned test-to-reference similarities to reflect label agreement.

2) Collaborative Inference:

  • Collaborative Inference applies the embedding learned during Collaborative Learning to predict labels for unseen samples.The reference set may be the same dataset used for Collaborative Learning.Source: 2) Collaborative Inference · p. 3
  • Training the embedding to match cross-batch similarities with label-agreement matrices motivates label prediction from test-to-reference similarities.The reference labels are known, while test labels are unknown.Source: 2) Collaborative Inference · pp. 3–4
  • Algorithm 2 summarizes the binary Collaborative Inference procedure.Source: 2) Collaborative Inference · p. 4

B. Collaborative Inference and Bayes Classifier

Collaborative Inference can be interpreted as a kernel approximation to the Bayes classifier. Its learned similarity function supplies the kernel used for Parzen-window estimates of class-conditional densities.

B. Collaborative Inference and Bayes Classifier

C. DKL with Collaborative Learning

The method replaces cosine similarity with an exponential base kernel using Mahalanobis distance, following Deep Kernel Learning. It learns the embedding and metric matrix from data.

C. DKL with Collaborative Learning

  • The base kernel uses Mahalanobis distance instead of cosine similarity, following the Deep Kernel Learning formulation.The distance is placed inside an exponential base kernel and rescaled to range from −1 to +1 to match the target matrix.Source: C. DKL with Collaborative Learning · p. 4
  • The metric matrix M is positive semidefinite and defined through a Cholesky factorization as M = LᵀL.L is a lower-triangular matrix of real values.Source: C. DKL with Collaborative Learning · p. 4
  • Both the embedding φθ and metric matrix Mθ are learned directly from data using neural networks.The paper compares this kernel with cosine similarity for CLaI multiclass prediction in Section IV-C.Source: C. DKL with Collaborative Learning · p. 4

D. Extending CLaI to Multiclass classification

The paper extends CLaI from binary classification to multiclass classification using a one-versus-rest strategy. Collaborative Learning conditions the embedding on a class and treats some already-separated negative samples as unknown during later stages.

D. Extending CLaI to Multiclass classification

  • CLaI is extended to multiclass classification by decomposing the task into one-versus-rest binary tasks.The section identifies this as a common approach to multiclass problems.Source: D. Extending CLaI to Multiclass… · p. 4

1) Collaborative Learning:

  • Class conditioning adds k to the embedding function φθ(X, k), creating class-specific mappings for the one-versus-rest strategy.The figure depicts multiclass classification proceeding in stages with different class conditioning.Source: 1) Collaborative Learning · p. 4, Figure 1 · p. 5
  • For a given class k, the target matrix assigns +1 when a sample label equals k and −1 otherwise.The similarity matrix is computed between embeddings from batches r and t.Source: 1) Collaborative Learning · p. 4
  • Samples in the same negative class are assigned zero and treated as unknown, reserving them for the next class conditioning.This is intended to avoid reclustering classes that have already been separated.Source: 1) Collaborative Learning · pp. 4–5

2) Collaborative Inference:

Collaborative Inference uses test samples and cross-similarity to produce predictions. For multiclass classification, it computes predictions for each class, applies softmax, and selects the most confident class.

2) Collaborative Inference:

  • Collaborative Inference predicts labels for test samples by leveraging cross-similarity.The test samples are arranged as a matrix X.Source: 2) Collaborative Inference · p. 5
  • The one-versus-rest approach computes a prediction f_k for each class.These classwise predictions are assembled into F = [f₁, …, f_C].Source: 2) Collaborative Inference · p. 5
  • The method applies softmax to F and selects the class with the most confident prediction.The full collaborative inference algorithm is referenced in the section.Source: 2) Collaborative Inference · p. 5

IV. EXPERIMENTS AND RESULTS

The experiments include multiclass classification datasets and a MIMIC-IV sepsis-detection setting. The section specifies dataset processing and identifies a classification-performance table and latent-space visualization.

A. Datasets and Preprocessing

  • MIMIC-IV vital signs from patients diagnosed with sepsis are used to detect sepsis events.The experiments extract multivariate recordings and corresponding sample labels using a previously described preprocessing procedure.Source: A. Datasets and Preprocessing · p. 5
  • CIFAR10 has 60,000 32×32 color images across ten categories, split into 50,000 training and 10,000 test images.Training and testing data are normalized separately using the dataset mean and standard deviation.Source: A. Datasets and Preprocessing · p. 5

IV. EXPERIMENTS AND RESULTS

  • The classification-performance table covers PathMNIST, CIFAR10, and SleepEDF.The supplied table caption identifies its scope but does not give performance values.Source: Table I · p. 6

A. Datasets and Preprocessing

  • The PathMNIST dataset consists of about 107,180 28×28 images of colorectal cancer tissue slides.PathMNIST is described as part of MedMNIST, a collection of biomedical images for classification.Source: A. Datasets and Preprocessing · pp. 5–6
  • SleepEDF experiments use 78 recordings split 80/20 for training and testing, with recordings segmented into non-overlapping 30-second windows.The preprocessing merges N3 and N4 into N3 and drops movement and unscored labels, leaving five stages.Source: A. Datasets and Preprocessing · p. 6

B. Multiclass Classification

CLaI demonstrates multiclass prediction across image and sleep datasets, while its Mahalanobis variant converges faster than cosine-based CLaI. In an auxiliary comparison, CLaI and Gaussian-process latent signals behave differently around events.

B. Multiclass Classification

  • CLaI supports multiclass prediction on CIFAR-10, PathMNIST, and SleepEDF, with lower performance on SleepEDF than on the other evaluated datasets.The lightweight ResNet-18 embedding is presented as a proof of concept, not a state-of-the-art model.Source: B. Multiclass Classification · p. 6

C. Ablation of the learned similarity function

D. Connection to kernel methods

2) Calibration of CLaI in sepsis prediction:

In sepsis prediction, CLaI-M has the lowest ECE among the compared CLaI variants, while CLaI’s framework connects learned representations with kernel alignment and Bayesian classification. The paper identifies multiclass computational complexity as its principal limitation and proposes reducing that cost as future work.

2) Calibration of CLaI in sepsis prediction:

  • CLaI-M has the lowest ECE among the compared CLaI variants, despite slightly lower performance across other metrics; CLaI also outperforms the Student-t mixture model in performance and calibration.The evaluation reports ECE, AUROC, AUPRC, and F1.Source: Table III · p. 7, 2) Calibration of CLaI in sepsis… · p. 7

V. DISCUSSION

  • Collaborative Learning aligns an embedding’s induced similarity with a label-derived target kernel, while Collaborative Inference has the form of a Parzen-estimated kernel Bayes classifier.Reference-class proportions estimate class priors, and within-class kernel-similarity sums estimate class-conditional densities.Source: §V · pp. 7–8
  • Across multiclass experiments, learned embeddings cluster same-class samples and keep related classes closer, suggesting alignment can capture relationships between classes.The discussion gives vehicles, four-legged animals, and blue-background classes as examples.Source: §V · p. 8
  • The Mahalanobis formulation improves multiclass performance, convergence speed, and sepsis calibration versus cosine similarity, while retaining kernel alignment.The discussion contrasts its Gaussian kernel’s infinite-dimensional feature space with cosine similarity’s finite-dimensional representation.Source: §V · p. 8
  • CLaI and Gaussian-process classification both use similarity-weighted reference contributions, but CLaI aligns to a label-derived target rather than maximizing marginal likelihood.The discussion positions CLaI as complementary to GP- and DKL-based methods.Source: §V · p. 8

VI. LIMITATIONS

  • Multiclass learning costs O(T · C · M^2), more than binary learning’s O(T · M^2), so the experiments use lightweight embeddings as proof of concept.The expressions omit per-pair cost; storage complexity is unaffected by the number of classes.Source: §VI · p. 8

VII. CONCLUSION AND FUTURE WORK

  • The experiments establish CLaI as a kernel-alignment method and provide a proof of concept for multiclass classification.Future work includes reducing multiclass computational cost, extending alignment to unsupervised learning, and exploring case-based interpretability.Source: §VII · pp. 8–9
Loading 2609.06474v2…