PaperlayerView exact revision on arXiv

Source-linked AI summary

Dataset Meta-Learning from Kernel Ridge-Regression

Timothy Nguyen, Zhourong Chen, Jaehoon Lee

arXiv:2011.00050v3cs.LGstat.ML

TL;DR

Machine-learning datasets are usually fixed, leaving open how to construct smaller or corrupted datasets that preserve downstream model performance. The paper defines ϵ-approximation and introduces KIP and LS to learn such datasets. KIP achieves one- or two-order-of-magnitude compression for KRR while supporting state-of-the-art distillation results and corrupted-data applications.

  • Problem

    Existing methods face scalability and privacy challenges because accurate kernel or instance-based prediction can require large support datasets of original examples.

  • Method

    The paper defines ϵ-approximate datasets and introduces KIP, which optimizes support datasets through kernel ridge-regression, alongside LS for direct label optimization.

  • Results

    KIP improves KRR sample efficiency by one or two orders of magnitude and achieves state-of-the-art results for KRR and neural-network dataset distillation.

  • Takeaways & Limitations

    Learned datasets can be substantially compressed or corrupted while retaining high performance across KRR and neural-network training settings.

  • Takeaways & Limitations

    Computational limits restricted experiments to shallower and less diverse architectures, with deeper architectures left for future work.

Abstract

from arXiv · show

One of the most fundamental aspects of any machine learning algorithm is the training data used by the algorithm. We introduce the novel concept of $ε$-approximation of datasets, obtaining datasets which are much smaller than or are significant corruptions of the original training data while maintaining similar model performance. We introduce a meta-learning algorithm called Kernel Inducing Points (KIP) for obtaining such remarkable datasets, inspired by the recent developments in the correspondence between infinitely-wide neural networks and kernel ridge-regression (KRR). For KRR tasks, we demonstrate that KIP can compress datasets by one or two orders of magnitude, significantly improving previous dataset distillation and subset selection methods while obtaining state of the art results for MNIST and CIFAR-10 classification. Furthermore, our KIP-learned datasets are transferable to the training of finite-width neural networks even beyond the lazy-training regime, which leads to state of the art results for neural network dataset distillation with potential applications to privacy-preservation.

1 INTRODUCTION

The paper reframes datasets as objects that can be optimized for downstream learning objectives. It introduces ϵ-approximation, KIP, and LS to obtain compact or corrupted datasets that preserve model performance.

  • Dataset design is optimized directly against a learning objective rather than treated as fixed training input.
  • ϵ-approximation asks which constrained datasets produce models approximately equivalent to a model trained on a specific dataset.
  • KIP is a meta-learning algorithm for obtaining ϵ-approximating datasets in supervised image learning.
  • The work reports state-of-the-art results for KRR and neural-network dataset distillation, including compressed and corrupted datasets.
  • 10 images can outperform hundreds or thousands for kernel ridge regression, with one- or two-order-of-magnitude sample-efficiency improvements.

2 SETUP

The setup defines dataset approximation through model-loss agreement, then frames compression and privacy as questions about the size and information content of alternative datasets.

  • A labeled dataset is represented as datapoints paired with label vectors, written D = (X, y).
  • Weak and strong ϵ-approximation compare trained models through expected loss rather than direct dataset similarity.
  • The compression question asks for the minimum size of an ϵ-approximating dataset at fixed ϵ.
  • The privacy question asks whether an ϵ-approximating dataset can satisfy a mutual-information bound relative to the original data distribution.
  • The compression ratio |D|/|D̃| is a heuristic measure because ϵ is not precisely quantified.
  • A ρ-corrupted dataset has at least a ρ fraction of each datapoint’s coordinates independent of the input dataset.

3 KERNEL INDUCING POINTS

KIP learns a small support dataset by optimizing kernel ridge-regression loss on target data, while LS directly solves for labels with fixed support inputs.

  • KIP seeks a small dataset whose trained model approximates the model obtained from a target dataset.
  • KIP optimizes support inputs, and optionally labels and kernel hyperparameters, through a first-order meta-learning procedure.
  • Using kernel families can produce datasets that approximate multiple kernels, improving robustness for neural-network training.
  • KIPρ creates ρ-corrupted datasets by initializing selected support coordinates with zeros or noise and not updating them.
  • LS minimizes the KRR loss directly over support labels while holding support inputs fixed, exploiting the loss’s quadratic form.

4 EXPERIMENTS

Experiments evaluate KIP and LS for dataset compression, kernel transfer, and transfer to neural-network training on MNIST, Fashion-MNIST, and CIFAR-10. Learned datasets achieve strong performance, including under corruption, while computational limits constrain architecture scale.

  • Experimental scope: Experiments cover compression, kernel-to-kernel transfer, and kernel-to-neural-network transfer on MNIST, Fashion-MNIST, and CIFAR-10.LS experiments include Fashion-MNIST; the datasets are evaluated across kernels and neural-network settings.
  • Single-kernel results: KIP achieves state-of-the-art KRR results on MNIST and CIFAR-10 for RBF and FC1 kernels in both accuracy and images required.The learned datasets rival deep convolutional networks on MNIST, exceeding 99.2% accuracy.
  • Single-kernel results: 10 or 100 KIP images perform on par with tens or hundreds times more natural images, yielding a compression ratio of one or two orders of magnitude.This comparison is reported for RBF and FC1 kernels.
  • Kernel-to-neural-network results: FC1 trained on KIP images outperforms prior CIFAR-10 neural-network distillation methods despite their deeper, more expressive architectures.On MNIST, KIP also outperforms some deeper-architecture baselines.
  • Kernel-to-kernel results: Using more targets improves LS performance, and all possible targets produce roughly one-order-of-magnitude compression at intermediate support sizes.The result is reported across FC and Myrtle-5/10 kernels on the three evaluated datasets.
  • Kernel-to-kernel results: KIP datasets trained with random sampling of six kernels transfer better on average than datasets trained with individual kernels.The transfer study uses FC and convolutional layers of depths 1–3.
  • Kernel-to-kernel results: FC1 and Myrtle-10 label-solving transfer is highly robust on CIFAR-10, with negligible performance differences between transferred and natural labels.Labels learned using either kernel can be used to train with the other.
  • Robustness studies: Corrupted KIP images generally transfer well to neural networks and often outperform uncorrupted natural images.Corruption sets a random fraction of pixels to uniform noise between −1 and 1.

5 RELATED WORK

The paper situates KIP among dataset compression, distillation, inducing-point, kernel-approximation, and neural-network-kernel methods. Unlike standard low-rank approximations, KIP optimizes support datasets for loss rather than reconstructing kernel matrices.

  • Coresets: KRR-based subset selection compresses datasets through coresets or related approximations.Borsos et al. is identified as a KRR-based dataset-subselection approach, while coresets are presented as a classical compression strategy.
  • Neural network approaches to dataset distillation: Neural-network dataset distillation learns input images or labels through gradient-based meta-learning and gradient matching.The cited approaches include hyperparameter-based image learning, label distillation, and gradient matching for condensing training sets.
  • Inducing points: KIP adapts the inducing-point idea by minimizing validation mean-square loss without requiring a probabilistic marginal-likelihood framework.The paper distinguishes KIP from Gaussian-process inducing-point methods by its validation-loss objective.
  • Low-rank kernel approximations: KIP produces low-rank support-support and target-support kernel matrices, but need not approximate the original matrices because it optimizes task loss.This distinguishes KIP from common low-rank kernel approximation methods.
  • Neural network kernels: Neural-kernel correspondences motivate using KRR to study wide-neural-network training and Bayesian inference.The paper relies on the correspondence between infinitely wide neural networks and kernel methods, including neural kernels.
  • Instance-Based Encryption: Sign corruption provides a related approach for encrypting individual images, whereas Figure 3 concerns transfer of KIP-learned images to finite neural networks.The figure compares natural and KIP-learned images across clean and 90% corrupted settings.

6 CONCLUSION

The paper concludes that KIP and LS meta-learn compressed or corrupted datasets with strong KRR and neural-network results. Its approximation notion allows algorithm variation but restricts approximating datapoints to the original shape, and its loss may differ from training loss.

  • 6 CONCLUSION: KIP and LS achieve state-of-the-art KRR and neural-network dataset-distillation results using shallow fully connected and purely convolutional models.The conclusion emphasizes that these results hold despite the limited expressiveness of the architectures.
  • 6 CONCLUSION: Highly corrupted datasets can match or exceed natural-image performance, motivating potential privacy-preserving machine-learning applications at scale.The conclusion presents privacy preservation as a potential application rather than an established deployment result.
  • 6 CONCLUSION: The approximation definition permits different learning algorithms, while requiring them to remain comparable for the dataset comparison to be meaningful.This flexibility accommodates preprocessing changes such as rescaling, but the paper leaves the algorithm class specification to each use case.
  • 6 CONCLUSION: Approximating datapoints must retain the original datapoints’ shape, making the definition stricter than returning extracted features.The restriction distinguishes dataset approximation from feature extraction.
  • 6 CONCLUSION: The dataset-comparison loss need not equal the losses optimized by the learning algorithms.For KRR, training minimizes mean-square loss while dataset comparison may use 0-1 loss.

B TUNING KIP

KIP tuning favors optimizing the full intended support set, class-balanced target sampling, image initialization, scale-invariant regularization, and rapid early training. Larger target batches help when train and test kernels agree but can increase overfitting when they differ or supports are large.

  • Sampling: Optimizing a support set directly at its intended size outperforms sampling smaller subsets from a larger optimized support set.The paper recommends sampling the entire support set during KIP training when optimizing KRR performance.
  • Sampling: Class-balanced target sampling benefits from larger batches when train and test kernels agree, whereas smaller batches reduce overfitting when they differ.Batch-size effects depend on kernel agreement between training and testing.
  • Initialization: Image initialization performs better than initializing support inputs with uniform noise and labels with mean-centered one-hot values.The comparison covers image initialization from a target subset versus noise initialization.
  • Regularization: Scale-invariant regularization normalizes regularization by support size and kernel rescaling, with λ = 10^-6 used in practice.The replacement regularizer uses the trace of the support-support kernel matrix.
  • Number of Training Iterations: After roughly one hundred iterations, KIP reaches more than 90% of final test accuracy across support sizes, kernels, and corruption factors.This rapid early learning is reported independently of the support size, kernel, and corruption factor.

C THEORETICAL RESULTS

The theoretical results characterize KIP’s convergence and approximation behavior for linear and general kernels. They connect optimized support representations to least-squares solutions, projection ranges, and strong ε-approximation guarantees.

  • Linear-kernel convergence: For generic initial conditions and sufficiently small λ > 0, gradient-descent KIP with a target dataset converges to a learned support dataset.The convergence theorem is stated for the linear-kernel setting.
  • Approximation guarantees: If the learned dataset has at least C points, its λ-ridge-regression classifier is also a least-squares classifier for the target dataset; uniqueness yields ε = 0.The theorem states this as a strong approximation with respect to (λ-RR, 0-RR) and mean-square loss.
  • Linear-kernel convergence: Gradient-flow KIP cannot converge to a non-global local minimum for sufficiently small λ under the stated fixed-initialization condition.The argument uses the transformed representation Φλ and excludes singular-value trajectories reaching the problematic locus.
  • Linear-kernel convergence: Generic gradient trajectories converge to finite solutions because the trajectories approaching the compactification’s infinite part form a nongeneric set.The convergence proof identifies the infinite-limit stable set as codimension one and therefore nongeneric.
  • Approximation guarantees: The difference between two linear models on the evaluation distribution is bounded pointwise by the spectral norm of their coefficient-matrix difference.This bound supplies the first step in the strong ε-approximation result.
  • Label solving: For λ = 0 and full-rank support and target feature matrices, the solved labels reproduce the 0-RR classifier’s predictions on the support set.The result identifies the label-solve solution with target-classifier predictions evaluated on Xs.
  • General kernels: For general kernels, the optimal KIP loss is determined by maximizing the attainable column space of KXtXs over support inputs.Trainable support labels make the transformed coefficients arbitrary, reducing optimization to the range of the target-support kernel map.

D EXPERIMENT DETAILS

The experiments specify kernel, optimizer, preprocessing, batching, initialization, and evaluation choices, while documenting compute and overfitting constraints. Performance can vary with support-set size and training configuration.

  • Optimization and preprocessing: KIP training used Adam, mean-centered 1-hot labels, class-balanced target batches, and dataset-specific learning rates of 0.01 for MNIST and 0.04 for CIFAR-10.Augmentation included flips, shifts, and rotations up to 10 degrees.
  • Architectures: The experiments used ReLU activations, while convolutional layers used 3 × 3 filters with stride 1 and same padding.These choices applied to both neural kernels and neural networks.
  • Compute limitations: Neural kernel computation scales linearly with depth, quadratically with convolutional image pixels, and quartically with pooling-layer pixels.These costs limited experiments to shallow kernels, small convolutional support sets and target batches, and few learned images with pooling.
  • Parameterization: Kernel parameterization had limited effects under scale-invariant KRR regularization relative to support-set size and learning rate.Experiments mixed NTK and standard parameterizations for variety.
  • Batching and overfitting: Large support sets can overfit when target batches are too large, leading the authors to exclude some 5K and 10K RBF CIFAR-10 entries.The authors note that a validation stopping criterion would require reducing the target dataset from the full training set.
  • Evaluation: The authors evaluated neural-network test accuracy across 5 random initializations and reported means and standard deviations.KRR experiments used 5 image-subset initializations, 5 checkpoints, and 25 evaluations.

E.1 KERNEL BASELINES

KIP datasets are compared with natural-image baselines across kernels and datasets, including transfer across kernel families. The reported results show strong compression and robustness, with average-kernel training improving cross-kernel performance.

  • Baselines: Random-subset baselines report accuracy on MNIST and CIFAR-10 with standard deviations over 20 resamplings.These tables provide the natural-image reference comparisons for the kernel experiments.
  • Kernel baselines: 10 KIP-learned images for RBF and FC1 achieved performance comparable to several thousand natural images, yielding compression ratios over 100.The compression ratio narrows as support size approaches the training-set size.
  • Cross-kernel transfer: Across FC1-3 and Conv1-3 kernels with NTK and NNGP variants, datasets sampled uniformly across all six kernels produced an overall performance boost.Evaluation also swapped FC and convolutional kernel widths at test time as a robustness check.
  • CIFAR-10: KIP datasets fit to RBF or FC1 outperformed CIFAR-10 baselines using many more images.Table A4 reports test accuracy comparisons for FC and RBF kernels.
  • Label transfer: Label-solve labels transferred between Myrtle-10 and FC1 without meaningful differences from natural labels across target counts and support sizes.The reported stability indicates that label-solve gains may be kernel-specific while the labels remain transferable.

E.3 KIP TRANSFER TO NEURAL NETWORKS AND CORRUPTION EXPERIMENTS

KIP-learned images transfer to finite-width neural networks and remain effective under severe pixel corruption. Comparisons vary dataset size, corruption, dataset type, and loss, with KIP often outperforming natural images.

  • Benchmark comparisons: Tables A5 and A6 compare KIP transfer with neural-network baselines on MNIST and CIFAR-10, using 90% noise corruption for the corrupted condition.The KIP images were learned with FC1-3 and Conv1-2 kernels.
  • Experimental factors: The appendix evaluates KIP and natural images on FC1 using mean-square and cross-entropy losses while varying dataset size and corruption.The MNIST tables cover 1K, 5K, and 10K image datasets; the CIFAR-10 tables report both losses.
  • MNIST: 90.9% MNIST test accuracy came from 1K KIP images with augmentation and 90% corruption, compared with 91.3% for 1K natural images and 75.0% for corrupted natural images.The example uses cross-entropy loss with an FC1 neural network.

F EXAMPLES OF KIP LEARNED SAMPLES

The visual examples compare KIP-learned samples with natural images for MNIST and CIFAR-10 under clean and 90% noise-corrupted conditions.

  • MNIST: MNIST examples show 100 KIP-learned images beside natural images, with separate rows for 0% and 90% noise corruption.The learned samples occupy the left side and natural samples the right side.
  • CIFAR-10: CIFAR-10 examples show 100 KIP-learned images beside natural images, again contrasting 0% and 90% noise corruption.The learned samples occupy the left side and natural samples the right side.
Loading 2011.00050v3…