PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-Efficient Classifier Pretraining Through Diffusion Models

Qinyu Li, Yee Whye Teh, Chris Holmes, Joel Jaskari

arXiv:2610.23289v1cs.LG

TL;DR

Limited labelled data can leave classifiers with many representations that fit the observed labels, and additional unlabelled data may be scarce or costly. The paper proposes DiffPT, a diffusion-based way to pretrain classifier backbones on the available dataset. It reports strong results across time-series and medical-image tasks, while SimCLR performs better for image classification when discriminative signals cover most of the image area.

  • Problem

    With limited labelled data, many representations can fit the observed labels, and the classification objective alone provides little guidance toward representations that generalize.

    Source: §1 · p. 1

  • Method

    DiffPT uses diffusion-model pretraining to improve a classifier backbone’s representation without requiring domain-specific augmentation strategies.

    Source: §1 · p. 2, §2 · p. 2, §2.1 · p. 2

  • Results

    DiffPT performs strongly across time-series and medical-image classification, while SimCLR outperforms it for image classification when discriminative signals cover most of the image area.

    Source: §6 · p. 7

  • Takeaways & Limitations

    DiffPT is an effective option for limited classification datasets, especially when the classification features are poorly understood.

    Source: §6 · p. 7

    SimCLR is sensitive to augmentation choices, requiring extensive hyperparameter search to select augmentations and their magnitudes for a domain.

    Source: §2.2.1 · p. 3

Abstract

from arXiv · show

Machine learning with limited data is susceptible to overfitting, especially with modern deep neural networks. Prior to the introduction of AlexNet, reconstruction-based pretraining with auto-encoder and denoising auto-encoder were found to be beneficial, partially due to the limited size of the training sets at the time. Motivated by these traditional approaches, we propose DiffPT, a simple diffusion modeling approach to pretrain a given classifier backbone for limited domain-specific training data. We experiment on time-series data, standard natural-image benchmarks, and medical imaging datasets, and contrast the diffusion approach to a multitude of modern and traditional un/self-supervised methods. While contrastive pretraining performs better on CIFAR-10 and Tiny ImageNet in low-data regimes, DiffPT is particularly effective on medical imaging and time-series tasks, where it outperforms self-supervised baselines across a range of low-data regimes. Our results suggest that DiffPT is a simple and effective approach for data-efficient classifier pretraining on different modalities.

1 Introduction

Limited labelled data can leave classifiers with many equally plausible representations, while existing pretraining approaches may depend on large datasets or domain-specific augmentation. The paper introduces DiffPT, a diffusion-based classifier-pretraining method, and reports that it matches strong augmentation-based self-supervised methods across modalities with fewer computational resources.

  • With scarce labels, many representations can fit observed labels equally well, leaving the classification objective with little guidance toward representations that generalize.The paper identifies this as a particular concern for high-dimensional data such as medical images and physiological signals.Source: §1 · p. 1
  • Earlier self-supervised methods can require very large unlabelled datasets and carefully designed augmentations, while non-imaging modalities lack a standardized application blueprint.ImageNet transfer learning may also be less useful when the downstream modality differs from natural images, and supervised pretraining depends on large labelled datasets.Source: §1 · pp. 1–2
  • DiffPT trains representations across a continuum of noise levels and avoids requiring practitioners to specify domain-specific augmentation strategies.Unlike a denoising auto-encoder trained at a fixed corruption level, diffusion training is described as capturing coarse global structure and fine local detail.Source: §1 · p. 2
  • The experiments use the same dataset for self-supervised pretraining and supervised fine-tuning, without adding unlabelled data.DiffPT is benchmarked against training from scratch, SimCLR, DINO, auto-encoder, and denoising auto-encoder pretraining with matched architectures.Source: §1 · p. 2
  • The paper proposes DiffPT and compares it with multiple self-supervised pretraining methods across physiological signals, natural images, medical imaging, and different degrees of data scarcity.Source: §1 · p. 2
  • DiffPT matches strong augmentation-based self-supervised methods across modalities while requiring fewer computational resources.Source: §1 · p. 2

2 Methodology

This section presents DiffPT as self-supervised pretraining of a classifier backbone, then compares it with several approaches adapted to the same backbone. It describes diffusion-based noise prediction and the baseline objectives.

2.1 DiffPT

  • DiffPT uses the classifier backbone as a diffusion encoder for pretraining, then fine-tunes it for downstream classification.Source: §2.1 · p. 2, §2 · p. 2
  • Using the DDPM formulation, DiffPT trains the model to predict noise across corruption levels ranging from light perturbation to near-noise.Source: §2.1 · p. 2
  • The diffusion model adds a lightweight decoder and skip connections to the backbone; for time series, interpolation aligns decoder features with encoder stages.Source: §2.1 · p. 2
  • Because the diffusion encoder is used as a feature extractor, DiffPT applies the same augmentations during pretraining and classifier training.Source: §2.1 · pp. 2–3

2.2 Baseline methods

  • The baselines are adapted to pretrain the same backbone used by DiffPT.Source: §2.2 · p. 3

2.2.1 SimCLR

  • SimCLR maximizes similarity between augmented views of the same image and minimizes it between different images, but its augmentation choices require extensive domain-specific tuning.Source: §2.2.1 · p. 3

2.2.2 DINO

  • DINO trains a student to match a teacher updated by exponential moving average, using global and local augmented views; its augmentation settings affect representation quality.Source: §2.2.2 · p. 3

2.2.3 Auto-encoder approaches

  • The auto-encoder uses a backbone encoder and a decoder without skip connections, while the denoising auto-encoder reconstructs clean inputs corrupted with Gaussian noise (σ = 0.1).Source: §2.2.3 · pp. 3–4

3 Related Works

Prior work explores diffusion-based classification and representation learning through modified diffusion models and auxiliary encoders. DiffPT is related to Xiang et al. (2023), but uses the target classifier backbone as its diffusion encoder and focuses on small-data discriminative performance.

3 Related Works

4 Experiments

The experiments evaluate DiffPT and comparison methods across natural-image, time-series, and medical-imaging datasets. They use reduced-data protocols and dataset-specific models, training, and evaluation choices.

4.1 Datasets

  • The six benchmarks cover natural images (CIFAR-10, Tiny ImageNet), ECG and activity time series (PTB-XL, WISDM), and medical images (EyePACS, HAM10000), with dataset-specific tasks and splits.PTB-XL uses 100 Hz ECGs in a 5-diagnostic-superclass multilabel task; WISDM records accelerometer and gyroscope data from 51 subjects performing 18 activities.Source: §4.1 · p. 4

4.2 Evaluation protocols

  • Reduced-data experiments reuse the same training subset for pretraining and fine-tuning, while full-data results are also reported.The sampled fractions vary by dataset; test sets remain full, and results average three seeds with different subsets.Source: §4.2 · pp. 4–5
  • Validation subsets are reduced with training data, and pretrained and fine-tuned checkpoints use validation criteria suited to each method and task.Generative-model checkpoints use validation loss, SimCLR and DINO use frozen-feature k-NN accuracy (macro-averaged AUROC for PTB-XL), and fine-tuned checkpoints use accuracy or macro-averaged AUROC.Source: §4.2 · pp. 5–6

4.3 Implementation details

  • Image classifiers use ResNet-18 or ResNet-34 with a CIFAR-style stem, while time-series tasks use a one-dimensional ResNet adaptation.The CIFAR-style stem avoids reducing spatial resolution too rapidly on medical images with small lesions.Source: §4.3 · p. 6
  • DiffPT and comparison methods share an AdamW-based training recipe, but SimCLR and DINO use method-specific pretraining augmentations.Augmentation hyperparameters for SimCLR and DINO were searched on medical imaging, where localized discriminative features can be sensitive to augmentation choices.Source: §4.3 · p. 6
  • Class-weighted cross-entropy balances class contributions, with weights computed from each run’s training data and reduced-data subsample when applicable.For PTB-XL, the task uses binary cross-entropy over five superclasses with the positive term weighted by class counts.Source: §4.3 · p. 6

5 Results

The results compare pretraining methods across natural-image, time-series, and medical-image classification, then assess their training-step costs. DiffPT’s relative performance varies across tasks, while reconstruction-based approaches use fewer resources than contrastive and self-distillation methods.

5.1 Self-supervised pretraining results

  • On subsampled CIFAR-10 and Tiny ImageNet, SimCLR performs best and DiffPT ranks second; with full data, DINO leads the pretraining methods on both benchmarks.On full-data CIFAR-10, scratch is on par with DINO; on Tiny ImageNet, scratch outperforms pretraining, while DiffPT is on par with DINO.Source: §5.1 · p. 6, Table 1 · p. 6
  • On WISDM, DiffPT improves accuracy over other pretraining methods and training from scratch at 50% and 100% data; on PTB-XL, it leads pretraining methods at 5% and 10%.On full-data PTB-XL, pretraining methods perform similarly to training from scratch, except DINO achieves the best performance among them.Source: §5.1 · pp. 6–7, Table 2 · p. 7
  • On EyePACS, DiffPT performs best at 1%, 10%, and 100% data, while on HAM10000, SimCLR leads and DiffPT ranks second.All pretraining methods outperform training from scratch on HAM10000; on EyePACS, all except DINO substantially outperform scratch in AUROC, including with full data.Source: §5.1 · p. 7, Table 3 · p. 7
  • SimCLR’s strongest results occur on datasets where the classified object typically covers most of the image, whereas DiffPT avoids location-based priors and is consistent across modalities and tasks.The authors suggest that SimCLR’s multicropping is well matched to such images, while crops can capture different classes when medical biomarkers are highly localized.Source: §5.1 · p. 7

5.2 Computational complexity

  • DiffPT, auto-encoder, and denoising auto-encoder require 11–24% more memory and are approximately 10% slower than classifier training from scratch.Their modest overhead is attributed to an asymmetric encoder-decoder architecture with a simple decoder.Source: §5.2 · p. 7
  • SimCLR and DINO require around 100% more memory and have 50–60% lower throughput than the baseline classifier.Their multi-view strategies increase resource demands; DINO’s teacher-network forward pass further increases latency.Source: §5.2 · p. 7

6 Conclusion

The paper benchmarks five self-supervised approaches for classifier pretraining on limited datasets. It concludes that DiffPT performs strongly across time-series and medical-image tasks, while SimCLR outperforms it on image classification when discriminative signals cover most of the image.

6 Conclusion

  • Across experiments from time-series to medical-image classification, DiffPT shows strong performance, and the authors recommend it for limited classification data, especially when features are poorly understood.They note that it can be used without extensive hyperparameter tuning.Source: §6 · p. 7
  • SimCLR outperforms DiffPT on image classification when discriminative signals cover most of the image area.Source: §6 · p. 7
  • The authors identify applying DiffPT to pretrain classification models for multimodal data as a future direction.Source: §6 · p. 8

A Data augmentations

The augmentation design keeps the fine-tuning input distribution consistent across methods, while adding separate view-generating augmentations only for SimCLR and DINO pretraining. Choices are adapted by modality, with clinical-image transformations preserving diagnostic signal and time-series views using temporal crops, noise, and masking.

A Data augmentations

  • All methods use the same base augmentation during fine-tuning; only SimCLR and DINO add view-generating augmentation during pretraining.The separate views support objectives that require multiple stochastically transformed versions of each sample.Source: A Data augmentations · p. 11
  • Base augmentation uses random crops and horizontal flips for natural images, additional flips, rotations, and rescaling for medical images, and random fixed-length windows for time-series.The time-series window is resampled whenever a record is drawn.Source: A Data augmentations · p. 11
  • Medical-image view generation reduces color-jitter strength, disables grayscale, and raises the minimum crop scale to preserve diagnostic signal.These settings are fixed across training-set sizes.Source: A Data augmentations · p. 11
  • Time-series views use independent fixed-length temporal crops, additive Gaussian noise, and contiguous zero-masking across all channels; crops are not resized.The settings table reports window lengths in samples and durations.Source: A Data augmentations · p. 11, Table 7 · p. 12
  • For SimCLR and DINO, image-view settings specify crops, color jitter, and grayscale, with DINO also using multi-crop views.The tables identify crop scales, color-jitter strength, and grayscale probability; DINO uses two global crops and local crops.Source: A Data augmentations · p. 11, Table 6 · p. 12
Loading 2610.23289v1…