Source-linked AI summary
Soft-Label Dataset Distillation and Text Dataset Distillation
Ilia Sucholutsky, Matthias Schonlau
TL;DR
Dataset distillation seeks to replace large training datasets with a few synthetic samples, but prior methods used hard labels and were demonstrated only for images. This paper jointly learns soft labels and samples, extends distillation to sequential data including text, and reports improved image results and strong text-distillation performance. The approach remains bounded by shared initialization distributions and untested cross-architecture generalization.
Problem
Prior dataset distillation used hard labels and had only been shown for image data, limiting its demonstrated scope.
Method
The paper jointly learns synthetic samples and soft labels, and embeds sequential data into a continuous space to enable text distillation.
Results
Across multiple datasets, learnable labels increase distillation accuracy by up to 6%, while text distillation broadens the datasets and architectures supported.
Takeaways & Limitations
Soft labels can encode information from multiple classes, while extending distillation to text increases the types of datasets and architectures to which it can be applied.
Takeaways & Limitations
Experiments use network initializations from the same distribution, and the paper does not test whether one distilled dataset transfers across architectures.
Abstract
from arXiv · showhide
Dataset distillation is a method for reducing dataset sizes by learning a small number of synthetic samples containing all the information of a large dataset. This has several benefits like speeding up model training, reducing energy consumption, and reducing required storage space. Currently, each synthetic sample is assigned a single `hard' label, and also, dataset distillation can currently only be used with image data. We propose to simultaneously distill both images and their labels, thus assigning each synthetic sample a `soft' label (a distribution of labels). Our algorithm increases accuracy by 2-4% over the original algorithm for several image classification tasks. Using `soft' labels also enables distilled datasets to consist of fewer samples than there are classes as each sample can encode information for multiple classes. For example, training a LeNet model with 10 distilled images (one per class) results in over 96% accuracy on MNIST, and almost 92% accuracy when trained on just 5 distilled images. We also extend the dataset distillation algorithm to distill sequential datasets including texts. We demonstrate that text distillation outperforms other methods across multiple datasets. For example, models attain almost their original accuracy on the IMDB sentiment analysis task using just 20 distilled sentences. Our code can be found at $\href{https://github.com/ilia10000/dataset-distillation}{\text{https://github.com/ilia10000/dataset-distillation}}$.
1. Introduction
Dataset distillation reduces training data to a few synthetic samples, and this work extends it by learning soft labels and applying the approach to sequential data. The proposed SLDD improves MNIST performance while enabling broader dataset-distillation use.
- Scope: The paper extends dataset distillation beyond images to sequential data, while the work is organized around method development, empirical validation, and discussion.The text section addresses embedding discrete text into a continuous space before distillation.
- Dataset distillation: Dataset distillation creates synthetic samples that can train networks to the accuracy achieved on the original dataset.Prior work trained LeNet to 94% MNIST accuracy using 10 synthetic images with known initializations.
- Soft labels: Soft-label examples represent hard labels, probability-distribution labels, and unrestricted real-valued labels.Hard labels can be obtained from unrestricted soft labels using softmax followed by selecting the highest-probability class.
- Soft labels: Soft-label dataset distillation makes synthetic labels learnable instead of restricting them to one-hot vectors.A soft label can associate one image with multiple classes, unlike the original fixed hard-label formulation.
- Results: 2% increase over the state-of-the-art: SLDD exceeds 96% MNIST accuracy with 10 distilled images.The figure reports 96.13% accuracy for networks with fixed initializations.
2. Related Work
Related work frames dataset distillation as a dataset-level counterpart to knowledge distillation and distinguishes it from methods that select or generate prototypes. The paper’s contribution is to optimize synthetic samples and their label distributions together.
- Knowledge distillation: Dataset distillation parallels network distillation by compressing knowledge from large datasets into a small set of synthetic samples.Network distillation compresses large or multiple networks into a smaller network, whereas dataset distillation compresses datasets into synthetic samples.
- Learning from small data: Dataset distillation addresses learning from small data by generating synthetic examples optimized for rapid model training.This contrasts with datasets such as MNIST and CIFAR10, which contain thousands of training images per class.
- Dataset reduction: Prototype selection reduces datasets using subsets of the original training data, whereas prototype generation creates new samples for nearest-neighbor classification.Selection is limited by the available points, while generation relaxes that restriction.
- Soft labels: Soft-label dataset distillation removes the fixed-label restriction by optimizing label distributions simultaneously with synthetic samples.This modification applies to the dataset-reduction setting described in the related work.
- Generative methods: Unlike generative approaches that mimic the true data distribution, dataset distillation uses knowledge distillation as its objective.The cited comparison concerns GAN-based image generation.
3. Extending Dataset Distillation
The paper extends dataset distillation by jointly optimizing synthetic samples and soft labels, then adapts the approach to sequential data such as text. Soft labels provide finer-grained class separation and allow fewer distilled points, while text is embedded into a continuous space for gradient-based distillation.
- 3.2 Basic Approach: Dataset distillation reduces a training set to synthetic samples optimized to achieve the original model’s accuracy with minimal training.The basic approach learns synthetic data through a one-step loss objective and gradient descent.
- 3.1 Motivation: Soft labels combined with prototype generation provide finer-grained class separation than selecting or moving points alone.In the kNN visualization, three classes can be separated using only two points when locations and label distributions are jointly modified.
- 3.3 Soft Labels: Soft-label distillation makes each synthetic label a learnable class distribution instead of a fixed one-hot label.The labels are optimized alongside the synthetic samples using backpropagation and gradient descent.
- 3.4 Text and Other Sequences: Text dataset distillation embeds sentences into a continuous representation so the image-oriented gradient method can be applied to sequential data.The method uses pre-trained GloVe embeddings; distilled embeddings are later mapped to nearest dictionary words to form sentences.
- 3.5 Random initializations and multiple steps: The distilled-data procedure assumes a fixed network initialization, although sampling initializations from a restricted distribution can improve generalization across initializations.The paper also reports that multiple gradient-descent steps and training epochs improve distillation performance for image and text data, particularly with random initializations.
4. Experiments
The experiments evaluate SLDD and TDD using distillation metrics, image and text baselines, and fixed or random network initializations. SLDD improves image distillation results over DD, while TDD achieves strong performance with small distilled text datasets.
- Metrics: Distillation ratio compares accuracy after training on distilled samples with original accuracy, while distillation size d_A is the minimum sample count needed to reach ratio A%.The M-sample ratio is reported as r_M, where M is the number of distilled samples.
- Image data: fixed initialization: SLDD reaches 96.13% MNIST accuracy with 10 distilled images and 91.56% with only 5 under fixed initialization, versus 93.76% for DD with 10 images.The MNIST 90% distillation size is d_A = 5, with accuracy beginning to plateau after five images.
- Image data: fixed initialization: SLDD provides a 7.5% increase in 100-sample distillation ratio on CIFAR10 over DD under fixed initialization.The passage also reports this as a 6% increase in distillation accuracy.
- Image data: random initialization: 82.75 ± 2.75% MNIST accuracy is achieved by SLDD with 100 images under random initialization, compared with 79.5 ± 8.1% for DD.On CIFAR10 with 100 images, SLDD yields a 3.8% increase in distillation ratio, corresponding to a 3% accuracy increase.
- Text data: TDD reaches an 89.88% distillation ratio on IMDB with 2 distilled sentences under fixed initialization and 79.96% with 2 sentences under random initialization.For the 50-class TREC50 task, TDD reaches a 79.86% ratio with 1000 sentences under fixed initialization.
5. Conclusion
The work broadens dataset distillation through learnable labels and compatibility with multiple neural architectures, while identifying limits in generalization across initializations and architectures. It also points to distilled datasets as a potential accelerator for compute-intensive search procedures.
- Up to 6% higher distillation accuracy across multiple datasets results from introducing learnable distilled labels.
- A single distilled dataset has not yet been tested across networks with different architectures, and both methods use initializations from the same distribution.Further investigation is needed to assess generalization across more varied initializations and architectures.
- The performance of dataset distillation depends on distilled-label initialization, with hard-label initialization outperforming random initialization.The authors suggest this may encourage earlier class separation, while the best treatment of similar classes remains unresolved.
- Dataset distillation works with CNNs, bi-directional RNNs, and LSTMs.Compatibility requires a twice-differentiable loss and gradients back-propagated to the inputs.
- Distilled datasets could speed up Neural Architecture Search and other compute-intensive meta-algorithms if they provide reliable performance proxies.The paper states that this could reduce search times by multiple orders of magnitude.