PaperlayerView exact revision on arXiv

Source-linked AI summary

EigDiff: Eigensample Diffusion - or how to significantly reduce training dataset size with marginal performance variation?

Andrei Jarca, Vlad Hondru, Florinel-Alin Croitoru, Radu Tudor Ionescu

arXiv:2610.20999v1cs.CVcs.AIcs.LG

TL;DR

Diffusion-model fine-tuning uses large datasets, while much efficiency work focuses on inference steps rather than training-data redundancy. EigDiff uses CLIP-space PCA eigenvectors to select exemplars and diverse samples. It generally matches or exceeds full-scale fine-tuning scores while reducing GPU-minutes and selection cost.

  • Problem

    Fine-tuning diffusion models uses exceptionally large datasets, while few existing works address training efficiency and some require manual or class-specific sample selection.

    Source: §1 · pp. 1–2

  • Method

    EigDiff embeds images with a frozen CLIP-based encoder, fits PCA on preferred-image embeddings, and selects a compact subset through variance-proportional exemplars and stratified sampling.

    Source: §3 · pp. 4–5, §5 · p. 10

  • Results

    EigDiff is best or second-best against four selection baselines in 12 of 14 cases, with 5.87×–13.95× lower GPU-minutes and 3.1×–1607× lower selection cost.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Across the tested datasets, diffusion backbones, and fine-tuning regimes, eigen-based selection generally matches or exceeds full-scale scores and surpasses random selection at equal budget.

    Source: §5 · p. 10

Abstract

from arXiv · show

Diffusion models are notoriously expensive to train, requiring large-scale datasets to learn how to gradually reverse the noise from images over multiple small denoising steps. Current approaches focus on distilling the number of diffusion steps, neglecting potential redundancies of the training data. In this paper, we explore the potential of harnessing eigensamples to train diffusion models on significantly smaller datasets, while preserving model performance. To mitigate the sensitivity to affine transformations of eigenimages, we harness the latent space of a pre-trained CLIP model. We next compute latent eigenvectors by performing principal component analysis (PCA) in CLIP space. Further, we employ eigenvectors as reference points to guide the selection of a representative training set comprising comprising both exemplars (images whose embeddings are closest to eigenvectors) and diverse images. Our comprehensive experiments with different diffusion models applied on multiple datasets indicate that our approach (EigDiff) can reduce training data by up to 14x and accelerate subset selection by up to 1607x, with marginal performance differences.

1 INTRODUCTION

EigDiff targets the large fine-tuning datasets that remain a training-efficiency bottleneck, unlike inference-step distillation. It selects representative samples using CLIP-space eigenvectors and reports similar performance with substantially less training time.

1 INTRODUCTION

  • Fine-tuning datasets remain exceptionally large, motivating representative sample selection to reduce their size.This addresses training data scale, whereas related work also targets the iterative inference process.Source: §1 · p. 1
  • Existing training-efficiency approaches require manual sample selection or class-specific selection tested only on labeled data, limiting real-world text-to-image applicability.EigDiff is presented as a fully automated alternative.Source: §1 · p. 2
  • EigDiff uses PCA eigenvectors in CLIP latent space to select aligned exemplars and stratified samples spanning varying degrees of alignment.The latent-space approach addresses the sensitivity of raw eigenimages to affine transformations.Source: §1 · p. 2, Figure 1 · p. 2
  • The reported eigenvector-based dataset distillation method yields similar performance under preference optimization and domain adaptation, including DPO and SFT, while reducing training time by 5.86× to 13.95×.Source: §1 · p. 3
  • The method is introduced as dataset distillation that selects representative training samples based on eigenvectors to reduce fine-tuning overhead.The paper evaluates its effectiveness and efficiency across datasets, models, and fine-tuning strategies.Source: §1 · p. 3

2 RELATED WORK

Related work reduces diffusion-model cost through inference-step distillation, architecture compression, and dataset selection. EigDiff addresses training-data redundancy with a lightweight selection procedure based on CLIP-space eigenvectors.

Time step and architecture distillation

  • Timestep distillation reduces inference steps, but still requires the full dataset and model during training, limiting fine-tuning with typical academic resources.Architecture distillation compresses models but requires teacher-model inference, an additional training overhead.Source: §2 · p. 3

Dataset distillation

  • Prior dataset-selection work for text-to-image diffusion has relied on manual labeling or computationally intensive selection, while much dataset distillation research focuses on supervised classification.Dai et al. (2023) use manual selection; Li et al. (2025a) use a classifier embedding, Gaussian scoring, and a reference diffusion model.Source: §2 · p. 4
  • EigDiff selects samples by similarity to eigenvectors computed in CLIP latent space, presenting a lightweight alternative to those prior selection approaches.The method targets data redundancy in diffusion fine-tuning rather than inference-step or architecture reduction.Source: §2 · p. 4

3 METHOD

EigDiff applies PCA to image embeddings and selects a compact training set in two stages: variance-proportional exemplars and stratified samples that broaden coverage across component scores.

Motivation

  • PCA on CLIP image embeddings yields eigenvectors that guide representative-image selection, avoiding raw eigenimages that are sensitive to affine transformations and lack natural-image detail.Exemplars capture dominant characteristics; stratified samples target diversity across similarity bins.Source: §3 · p. 4

Principal components selection

  • For DPO, PCA is fit on preferred-image embeddings so the components describe their dominant variation; SFT uses individual image embeddings.The resulting eigenvectors are sorted by eigenvalue, and components are selected using explained-variance thresholds.Source: §3 · pp. 4–5

Stage 1

  • Stage 1 selects the highest-similarity exemplars for components in the first variance-defined set, allocating its sample budget according to explained variance.For each component, the selected exemplars are its most similar samples.Source: §3 · p. 5

Sample scoring

  • DPO scores are signed chosen-minus-rejected alignment contrasts, while SFT scores are raw image alignment with a principal component.The DPO sign retains preference direction for the downstream objective.Source: §3 · p. 5

Stage 2

  • In Stage 2, the method uses principal components in B, sets total sample budget n2 = δ · n1, computes each component’s similarity scores, and partitions each score distribution into B evenlyspaced empirical quantile intervals.For each component and interval, it uniformly draws floor(ki/B) samples.Source: §3 · p. 5

Final training set

  • The final training set is the union of the Stage 1 exemplar set and the Stage 2 diverse sample set.The method selects images from the original training data rather than using eigenimages as training data.Source: §3 · pp. 4–5

4 EXPERIMENTS

The experiments evaluate EigDiff across four datasets, two diffusion models, and DPO and SFT fine-tuning, comparing performance and computational costs with full-scale and data-selection baselines. EigDiff generally outperforms random selection while reducing training and selection time, with qualitative comparisons also favoring EigDiff.

4.1 Datasets

  • The evaluation covers Pick-a-Pic, HPDv2, Rapidata, and out-of-domain ROCO Radiology, with preference-dataset filtering and dataset-specific training and test sizes.Pick-a-Pic and HPDv2 are subsampled to 100,000 training examples each; ROCO retains 42,050 training pairs after filtering.Source: §4.1 · pp. 5–6

4.2 Training Setup

  • The experiments compare pretrained and full-scale references, random selection, and four coreset methods under common budgets, using LoRA fine-tuning and domain-appropriate evaluation metrics.PickScore is used for in-domain datasets and BiomedCLIP for ROCO Radiology.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 Quantitative Results

  • 5.87–13.95× less fine-tuning time and 3.1–1607× faster selection are reported for EigDiff versus full-scale runs and other selection methods, respectively.Fine-tuning comparisons use equivalent epochs; selection timings use precomputed features.Source: §4.3 · p. 6, Table 2 · p. 7, Table 3 · p. 7
  • EigDiff beats random in every DPO case, leads SD1.5 on all datasets, and exceeds full-scale in three settings; PixArt-α outcomes vary.For PixArt-α, EigDiff is within 0.001 of full-scale on Pick-a-Pic and exceeds it on Rapidata and HPDv2.Source: §4.3 · p. 7, Table 4 · p. 8
  • EigDiff beats random in every SFT case and outperforms competitors in six of eight cases; SD1.5 leads all four datasets, while PixArt-α leadership varies.With PixArt-α, EigDiff leads on Rapidata and ROCO, while other selection methods lead on Pick-a-Pic and HPDv2.Source: §4.3 · pp. 7–8, Table 5 · p. 8
  • Stage 1 supplies the dominant preference signal, Stage 2 adds diversity for training stability, and the complete method outperforms random selection; B=4 and δ=6 yield the best overall results.Ablations also report τ1=0.6 and τ2=0.8 as ideal for Pick-a-Pic with SD1.5.Source: §4.3 · pp. 8–9, Table 6 · p. 9, Table 8 · p. 9

4.4 Qualitative Results

  • Across the reported DPO and SFT examples, EigDiff is frequently described as producing more prompt-aligned, higher-quality images with fewer hallucinations than comparison methods.The qualitative discussion includes text-to-image examples and medical images; on ROCO, full-scale has higher quantitative scores despite less favorable qualitative depictions.Source: §4.4 · pp. 9–10

5 CONCLUSION

EigDiff selects a compact preference-pair subset using PCA directions and a two-stage stratified procedure. Across the evaluated backbones, regimes, and datasets, it generally matches or exceeds full-scale scores while reducing training and selection costs.

5 CONCLUSION

  • The pipeline embeds preference pairs with a frozen CLIP-based encoder, fits PCA on preferred-image embeddings, then selects a compact subset through variance-proportional stratified stages.Pairs are scored by signed alignment with principal directions.Source: §5 · p. 10
  • EigDiff generally matches or exceeds full-scale fine-tuning scores, surpasses equal-budget random selection, and is best or second-best against selection baselines in 12 of 14 cases.It reduces GPU-minutes by 5.87×–13.95× and selection cost by 3.1×–1607× versus four selection baselines.Source: §5 · p. 10
  • Ablations indicate deterministic exemplars carry the dominant preference signal, while stratified secondary directions add non-redundant diversity and variance-proportional budgeting links the selection stages.The authors describe the selection cost as a single PCA over cached embeddings, making it a cheap, model-agnostic preprocessing step.Source: §5 · p. 10
  • The method relies on the embedding geometry of its encoder.Source: §5 · p. 10

Related papers

Loading 2610.20999v1…