PaperlayerView exact revision on arXiv

Source-linked AI summary

Mini-batch Sampling Strategies for Long-Tailed Image Classification: An Empirical Study on CIFAR-100-LT

Siyu Yuan

arXiv:2609.16365v1stat.MLcs.CVcs.LG

TL;DR

Long-tailed datasets leave rare classes underrepresented in mini-batches, raising the question of how sampling affects SGD learning. The paper compares four strategies in a bias–variance framework and controlled experiments; progressive sampling improves tail accuracy while overall accuracy remains indistinguishable from uniform sampling at the reported higher imbalance ratios.

  • Problem

    The paper asks how mini-batch sampling strategies affect SGD convergence and classification on long-tailed data, where uniform sampling can underrepresent rare classes.

    Source: §1 · p. 2, §2.2 · p. 4

  • Method

    The paper analyses four sampling strategies in a unified bias–variance framework and compares them in controlled ResNet-32 experiments on CIFAR-100-LT across three imbalance ratios.

    Source: §7 · p. 23

  • Results

    Progressive sampling is best on tail classes where they exist, while its overall accuracy is indistinguishable from uniform sampling at ρ = 50 and ρ = 100 when runs are paired by seed.

    Source: §7 · p. 23

  • Takeaways & Limitations

    The authors recommend linear-schedule progressive sampling as a default for long-tailed training and advise avoiding pure class-balanced sampling when the smallest classes are very small.

    Source: §7 · p. 23

    The findings are based on one architecture and dataset, with confirmation on larger benchmarks still needed.

    Source: §6.3 · p. 21, §7 · p. 23

Abstract

from arXiv · show

Real-world datasets often exhibit long-tailed class distributions, where a few head classes contain a large number of training samples while a large number of tail classes have only a few. The composition of each mini-batch, determined by the sampling strategy, governs which classes contribute to the stochastic gradient estimate, and therefore affects convergence behaviour and generalisation across the whole class spectrum. We provide a systematic theoretical and empirical comparison of four mini-batch sampling strategies for long-tailed image classification: uniform instance sampling, class-balanced sampling, square-root sampling, and progressively balanced sampling. We place all four in a unified bias-variance framework describing their effect on gradient estimation, which exposes the tension between unbiased optimisation of the empirical loss and fair representation of rare classes. We then evaluate them under controlled conditions using ResNet-32 on CIFAR-100-LT at three imbalance ratios (rho = 10, 50, 100), with every strategy sharing the same long-tailed subsets and initialisation within a seed. Progressive sampling improves tail-class accuracy by 25% relative to the uniform baseline at rho = 100 (13.5% versus 10.8%), consistently across all three seeds, while its overall accuracy is not distinguishable from that of uniform sampling given the seed-to-seed variation (40.0% versus 39.7%); the tail-class gain, not the overall gain, is the robust effect. At rho = 100, class-balanced sampling degrades accuracy on every class group, including the tail classes it is designed to help, which we attribute to overfitting caused by extreme oversampling of scarce data; at rho = 50 this failure is confined to head and medium classes. These results indicate that when rebalancing is applied during training matters as much as how much rebalancing is applied.

1 Introduction

Long-tailed class frequencies make mini-batch composition consequential: uniform sampling can leave rare classes with insufficient gradient signals. The paper frames sampling choice as a trade-off and contributes a unified analysis and controlled comparison of four strategies.

1 Introduction

  • Uniform mini-batches are dominated by head classes, leaving rare classes with insufficient gradient signals and poor tail-class performance.The paper asks how sampling strategies affect SGD convergence and classification performance on long-tailed data.Source: §1 · p. 2

1.1 Contributions

  • The paper provides a unified mathematical framework for characterising the bias and variance of mini-batch sampling strategies in long-tailed learning.It also analyses learning dynamics and trade-offs between head- and tail-class performance.Source: §1.1 · p. 2
  • It compares four sampling strategies on CIFAR-100-LT across three imbalance ratios and offers practical recommendations based on the findings.The strategies are uniform instance, class-balanced, square-root, and progressively balanced sampling.Source: §1.1 · pp. 2–3

2 Background and Related Work

Long-tailed recognition concerns class distributions in which a few classes contain most samples and many classes contain few. The background describes how uniform sampling underrepresents tail classes and situates the paper’s controlled sampler comparison among related approaches.

2.1 The Long-Tailed Recognition Problem

  • A long-tailed distribution has a few head classes containing most samples and many tail classes containing few.The imbalance ratio measures the largest-to-smallest class-size ratio.Source: §2.1 · p. 3

2.2 The Challenge of Uniform Sampling Under Imbalance

  • At ρ = 100, uniform sampling leaves a given tail class absent from roughly 94% of mini-batches, producing sparse, noisy gradients and prioritising head-class performance.For CIFAR-100-LT, the largest class has 500 samples and the smallest has 5.Source: §2.2 · p. 4

2.3 Related Work on Long-Tailed Learning

  • Related approaches include resampling, reweighting, and transfer learning; resampling can overfit minority classes or discard majority-class information.The paper’s class-balanced sampler selects a class uniformly, then samples an instance from that class.Source: §2.3 · p. 4
  • The four samplers were previously contrasted by Kang et al. [11]; this paper instead focuses on their bias–variance analysis and controlled CIFAR-100-LT comparison.Within each seed, strategies use identical long-tailed subsets and initial weights, and the study tracks head, medium, and tail accuracy during training.Source: §2.3 · p. 5

3 Methodology

The methodology treats sampling strategies as different weightings of class losses and compares their effects on gradient estimation. It describes four samplers and analyses the resulting bias–variance trade-offs, including progressive rebalancing over training.

3.1 Problem Setup and Notation

  • Different sampling strategies implicitly optimise different weightings of class losses, which the paper analyses through gradient-estimation bias and variance.The setup defines per-sample cross-entropy and the empirical class prior.Source: §3.1 · p. 5, §3 · p. 5

3.2.1 Strategy 1: Uniform Instance Sampling

  • Uniform instance sampling is unbiased for the empirical loss, but tail classes contribute to only about 6% of mini-batches at ρ = 100.Its expected class weight is proportional to class frequency, leaving tail parameters refined on relatively few steps.Source: §3.2.1 · p. 6

3.2.2 Strategy 2: Class-Balanced Sampling

  • Class-balanced sampling selects a class uniformly before sampling an instance, giving equal expected gradient contribution to each class but risking overfitting scarce tail examples.It is biased relative to uniform sampling and corresponds to the class-balanced loss.Source: §3.2.2 · pp. 6–7

3.2.3 Strategy 3: Square-Root Sampling

  • Square-root sampling weights class selection in proportion to the square root of class frequency, partially compensating for imbalance without equalising class contributions.Tail representation lies between uniform and class-balanced sampling, reducing overfitting risk relative to class balancing.Source: §3.2.3 · p. 7

3.2.4 Strategy 4: Progressively Balanced Sampling

  • Progressively balanced sampling transitions from uniform to class-balanced sampling over training, beginning with head-rich feature learning and later focusing on tail classes.The transition speed is controlled by γ.Source: §3.2.4 · p. 8

3.3 Theoretical Analysis: Bias-Variance Trade-offs

  • 470: under the stated variance model, class-balanced sampling inflates the tail-class gradient-variance contribution by roughly 470-fold relative to uniform sampling at ρ = 100.The estimate assumes equal within-class gradient variance, an assumption the authors say is violated in practice; they caution that the ratio concerns gradient noise, not proof of test-accuracy degradation.Source: §3.3 · p. 9
  • No static strategy minimises both bias and variance; progressive sampling lets bias grow over training while its predicted class-wise performance patterns are tested experimentally.Its early estimator is nearly unbiased when λ is near zero, while the bias term scales as λ².Source: §3.3 · pp. 9–10

4 Experimental Setup

The experiments use a controlled CIFAR-100-LT setup with ResNet-32 and shared data subsets and initialisation across sampling strategies within each seed. The section specifies the dataset, model, training schedule, and evaluation protocol.

4 Experimental Setup

  • The experiments follow the standard long-tailed recognition protocol [5] to support comparability with prior work.Source: §4 · p. 10

4.1 Dataset: CIFAR-100-LT

  • CIFAR-100-LT reduces training samples per class by exponential decay from a 500-sample maximum; evaluation retains 100 test images per class.Source: §4.1 · pp. 10–11
  • At ρ=10, all classes have at least 49 training samples, leaving the tail group empty under the consistent frequency grouping.Head classes have more than 100 samples, while medium classes have 20 to 100 samples.Source: §4.1 · p. 11

4.2 Model Architecture

  • ResNet-32 uses three residual stages, global average pooling, and a fully connected classifier, with approximately 0.47 million parameters.Source: §4.2 · p. 11

4.3 Training Configuration

  • Experiments use SGD with momentum and three seeds; within each seed, all strategies share the same long-tailed subset and model initialisation.The study reports mean and standard deviation across runs and evaluates overall and per-group accuracy on the balanced test set.Source: §4.3 · p. 11, Learning rate schedule · pp. 12–13

Learning rate schedule.

  • The learning rate warms from 0.02 to 0.1 over five epochs, then drops to 0.001 at epoch 160 and 10^-5 at epoch 180.Gradient norm clipping is set to 5.0; non-uniform samplers draw the training-set size with replacement, recomputing progressive weights each epoch.Source: Learning rate schedule · p. 12

5 Results and Analysis

Across three seeds, the study compares four mini-batch sampling strategies on CIFAR-100-LT at three imbalance ratios. Progressive sampling’s tail-class gains are consistent across seeds, while its overall-accuracy advantage over uniform sampling is not established.

5.1 Overall Performance Comparison

  • Progressive sampling records 57.6%, 44.1%, and 40.0% overall accuracy at ρ = 10, 50, and 100, but its lead over uniform is not a reliable ranking.Class-balanced sampling’s deficit versus uniform grows from 0.6 points at ρ = 10 to 4.2 at ρ = 50 and 5.7 at ρ = 100.Source: Table 4 · p. 13, §5.1 · pp. 13–14, §5.3 · p. 16

5.2 Paired Per-Seed Comparison

  • Progressive sampling’s tail-accuracy advantage holds for every seed, averaging +2.72 points at ρ = 100 and +2.37 points at ρ = 50; its overall advantage is not robust.There are only three seeds, so the authors report paired differences rather than p-values.Source: Table 5 · p. 14, §5.2 · p. 14

5.3 Head vs Medium vs Tail Analysis

  • At ρ = 100, progressive sampling reaches 13.5% tail accuracy versus uniform’s 10.8%, a 25% relative gain, while head accuracy is 64.9% versus 66.7%.At ρ = 50, progressive sampling also leads on tail accuracy, with 16.2% versus uniform’s 13.9%.Source: Table 6 · p. 15, §5.3 · p. 16
  • At ρ = 100, class-balanced sampling has lower tail accuracy than uniform (9.2% versus 10.8%) and head accuracy of 57.7% versus 66.7%; at ρ = 50, its tail gain is marginal.The authors associate the ρ = 100 result with oversampling classes containing only five training samples; at ρ = 50, class-balanced sampling loses 7.6 and 2.8 points on head and medium classes.Source: Table 6 · p. 15, §5.3 · pp. 15–16

5.4 Convergence Dynamics

  • During early training, progressive and uniform sampling lead class-balanced sampling; the epoch-160 learning-rate decay brings a 5–8-point accuracy jump, and accuracy plateaus after the epoch-180 decay.Class-balanced training loss eventually approaches zero despite lower test accuracy, which the authors interpret as overfitting on oversampled tail classes.Source: §5.4 · p. 17, Figure 1 · p. 17

5.5 Effect of Imbalance Ratio

  • As ρ rises, medium accuracy declines for every strategy, while progressive sampling remains highest; the tail group is empty at ρ = 10, so its plotted zero is not an accuracy.At ρ = 100, uniform head accuracy is 66.7%, and class-balanced head accuracy falls from 62.1% at ρ = 10 to 57.7%.Source: §5.5 · p. 18, Figure 2 · p. 18

5.6 Analysis of Progressive Sampling

  • Progressive sampling tracks uniform on head and medium classes early, then tail accuracy rises as λ grows; its tail trajectory is steadier than class-balanced sampling’s.The epoch-160 learning-rate decay coincides with λ ≈ 0.80, so the experiment cannot isolate their contributions to the tail-accuracy gain at that point.Source: §5.6 · p. 19, Figure 3 · p. 19

6 Discussion

The discussion frames sampling as a bias–variance trade-off and argues that when rebalancing occurs matters as much as how much rebalancing is used. It recommends progressive sampling when tail accuracy matters, while noting limits to the study’s scope and identifying directions for further work.

6.1 The Bias-Variance Trade-off in Practice

  • At ρ = 100, class-balanced sampling degraded every class group, with extreme oversampling of five-sample tail classes associated with memorisation rather than generalisable features.The discussion notes that the variance from extreme oversampling dominates in this regime.Source: §6.1 · p. 20
  • Progressive sampling can be interpreted as implicit curriculum learning: head-data-driven feature learning precedes fine-tuning for tail classes as λ increases.The proposed interpretation connects abundant head-class data to a useful initialisation for all classes through shared low- and mid-level representations.Source: §6.1 · p. 20
  • Square-root sampling is a time-invariant compromise between uniform and balanced sampling, but does not capture stage-dependent training needs as progressive sampling does.Its class probabilities are proportional to √n_k.Source: §6.1 · p. 20

6.2 Practical Recommendations

  • Progressive sampling is recommended when tail accuracy matters; when only aggregate accuracy matters, the results give no reason to prefer it over simpler uniform sampling.The reported tail-class advantage comes with an approximately 1–2-point head-class cost and no measurable overall-accuracy cost.Source: §6.2 · p. 20
  • Avoid pure class-balanced sampling when the smallest classes have only a handful of examples; n_min < 10 is a rough indication, not a threshold.At ρ = 100 it harmed all class groups, whereas at ρ = 50 it did not harm tail accuracy; the boundary is unresolved by two data points.Source: §6.2 · p. 21
  • The learning-rate decay coincides with late-stage progressive rebalancing, but their interaction is untested because the two schedules were never varied independently.The authors flag this as an open design question, not a validated recommendation.Source: §6.2 · p. 21

6.3 Limitations

  • The study is limited to ResNet-32 on synthetically imbalanced CIFAR-100-LT, cross-entropy loss, one progressive schedule, top-1 accuracy, and best-test-checkpoint selection.The paper also notes test-set optimism of 0.06–0.30 percentage points, with none of its conclusions depending on the checkpoint protocol.Source: §6.3 · pp. 21–22

6.4 Future Directions

  • Future work proposes adaptive transition schedules, combinations with complementary methods, larger-benchmark evaluation, and finite-sample analysis of weighted gradient estimation.The suggested combinations include decoupled training, feature-level augmentation, and specialised loss functions.Source: §6.4 · p. 22

7 Conclusion

Across CIFAR-100-LT experiments, progressive sampling reliably improves tail-class accuracy while overall accuracy remains indistinguishable from uniform sampling at severe imbalance. Class-balanced sampling can harm all groups when classes are extremely scarce, and sampler differences grow with imbalance severity.

7 Conclusion

  • Progressive sampling is best on tail classes where tails exist, with gains over uniform sampling in every random seed.Its overall accuracy is indistinguishable from uniform at ρ = 50 and ρ = 100, so the gain is a redistribution toward rare classes rather than a uniform improvement.Source: §7 · p. 23
  • Early uniform-like sampling builds representations from abundant head data before the transition redirects learning toward underrepresented classes.The authors interpret this as a two-phase learning dynamic.Source: §7 · p. 23
  • At ρ = 100, class-balanced sampling degrades every class group, including the tail; at ρ = 50, it still loses on head and medium classes but does not harm the tail.The severe-imbalance failure is attributed to oversampling classes with as few as 5 training samples and inflated tail-gradient variance.Source: §7 · p. 23
  • The strategy-performance gap widens from 1.6 points at ρ = 10 to 6.0 points at ρ = 100.The authors conclude that sampler selection matters more as imbalance severity increases.Source: §7 · p. 23
  • The authors recommend progressive sampling with a linear schedule (γ = 1) as a default and advise against pure class-balanced sampling when the smallest classes are very small.The conclusions are based on one architecture, one dataset, and three seeds; the authors call for confirmation on larger benchmarks.Source: §7 · p. 23

Code Availability

The repository provides code to reproduce the experiments, figures, and tables, including the dataset construction, samplers, training pipeline, and per-epoch logs.

Code Availability

  • The repository contains the CIFAR-100-LT construction, four samplers, the training pipeline, and per-epoch logs underlying the reported results.It also provides code to reproduce the experiments, figures, and tables.Source: Code Availability · p. 24

A Implementation Details

The appendix documents sampler implementation, training and evaluation procedures, and tables reporting overall and per-group results across seeds and imbalance ratios.

A.1 Sampling Strategy Implementation

  • Non-uniform strategies use replacement-weighted sampling; square-root sampling weights classes by sqrt(n_k), while progressive sampling interpolates between uniform and balanced class probabilities.Uniform sampling uses DataLoader shuffle instead.Source: §A.1 · pp. 27–28

A.2 Learning Rate Schedule

  • The learning-rate schedule uses linear warmup followed by decay at configured milestones.The implementation description specifies linear warmup and decay at milestones, with the listed milestones at epochs 160 and 180.Source: §A.2 · p. 28

A.3 Training Loop

  • The single-epoch training loop includes gradient clipping for stability.The loop returns average loss and accuracy for the epoch.Source: §A.3 · pp. 28–29

B.1 Per-Seed Overall Accuracy

B.2 Detailed Head/Medium/Tail Accuracy

B.3 Per-Seed Head/Medium/Tail Breakdown

C Best-Epoch versus Final-Epoch Results

Final-epoch checks preserve the robust tail-accuracy advantage of progressive over uniform sampling, while overall-accuracy ordering remains unreliable. The appendix also reports a small best-versus-final selection gap and illustrates schedules and convergence curves.

C Best-Epoch versus Final-Epoch Results

  • Best-epoch test accuracy exceeds final-epoch accuracy by 0.06–0.30 percentage points, with a mean gap of 0.16.The authors report that these gaps are an order of magnitude smaller than the strategy differences they interpret.Source: C Best-Epoch versus Final-Epoch Results · p. 32, Table 16 · p. 33
  • Progressive sampling beats uniform on tail accuracy for every seed at both ρ = 100 and ρ = 50 under final-epoch evaluation.Mean differences are +2.49 points at ρ = 100 and +1.96 at ρ = 50; sign agreement is 3/3 in both cases.Source: C Best-Epoch versus Final-Epoch Results · p. 33

D.1 Overall Accuracy Comparison

  • Overall-accuracy rankings remain unreliable: progressive and uniform exchange places at ρ = 50 under final-epoch evaluation, while class-balanced remains last at every ratio.The authors caution that progressive and uniform are not separable in overall accuracy at this sample size.Source: C Best-Epoch versus Final-Epoch Results · p. 33, Figure 4 · p. 34

D.2 Head/Medium/Tail Accuracy Bars

  • The head/medium/tail accuracy figure compares class-group performance at ρ = 100 and ρ = 50, with error bars showing standard deviation over three seeds.Source: Figure 5 · p. 34

D.3 Learning Rate and Progressive λ Schedule

  • The schedule figure shows learning-rate milestones at epochs 160 and 180 and a progressive λ(t) that rises from 0 to 1, reaching about 0.80 at the first milestone.The plotted λ(t) is scaled by ×0.1 for visual clarity.Source: Figure 6 · p. 35, §D.3 · p. 35

D.4 Convergence Curves for ρ = 50 and ρ = 10

  • At ρ = 10, all four strategies converge to similar final accuracy, consistent with the mild imbalance.The ρ = 50 figure plots test accuracy and training loss with standard-deviation bands over three seeds.Source: Figure 7 · p. 35, Figure 8 · p. 36

D.5 Per-Group Convergence Curves

E Dataset Class Distribution

CIFAR-100-LT class frequencies form an exponential long tail whose severity varies with the imbalance ratio; the section also specifies frequency-group boundaries and reports group sizes.

E Dataset Class Distribution

Related papers

Loading 2609.16365v1…