PaperlayerView exact revision on arXiv

Source-linked AI summary

Pretraining and Distillation Matter More Than Architecture Family for Label-Free Single-Cell Classification

Philip Graemer, Giuseppe Di Caprio

arXiv:2609.09863v1cs.CV

TL;DR

Conflicting microscopy results leave CNN–transformer comparisons difficult to interpret as stable architecture-family effects. In a controlled LIVECell benchmark, pretraining improved performance, while pretrained CNNs and transformers showed no evidence of residual family-level separation.

  • Problem

    Microscopy benchmarks report divergent CNN–transformer comparisons, and transformer performance depends strongly on pretraining and training recipe, complicating interpretation of stable architecture-family effects.

    Source: §1 · p. 4

  • Method

    The study compares CNN and transformer backbones on LIVECell with source-image-disjoint splits and matched optimisation, augmentation, and evaluation protocols, while testing pretraining and distillation.

    Source: §2.1 · p. 6, §1 · p. 6

  • Results

    Pretraining raised EfficientNet-B5 macro-F1 by 4.3 points versus random initialisation; pretrained comparisons showed no evidence of residual CNN superiority.

    Source: §2.2 · p. 8, §3.2 · p. 15

  • Takeaways & Limitations

    Council distillation let compact EfficientNet-B0 students retain approximately 85–89% of direct councils’ improvement over the undistilled baseline with one forward pass, at substantially lower inference cost.

    Source: §3.3 · p. 18

    Because the benchmark uses one controlled public dataset of cleanly segmented, background-reduced crops, performance under cross-instrument, cross-laboratory, or cross-modality conditions remains uncharacterised.

    Source: §3.5 · p. 19

Abstract

from arXiv · show

Choosing a deep learning architecture for label-free single-cell classification remains an open question, with microscopy benchmarks reporting conflicting conclusions about CNNs versus transformers. We present a controlled benchmark on LIVECell phase-contrast microscopy data using source-image-disjoint train/validation/test splits to prevent parent-image leakage and matched optimisation, augmentation, and evaluation protocols across EfficientNet, Vision Transformer (ViT), and EVA-02 models. This allows the effects of architecture, pretraining, fine-tuning, tokenisation, and distillation to be disentangled. We find that the previously reported CNN advantage is largely explained by pretraining rather than architecture: the smallest pretrained model outperforms the strongest model trained from scratch despite far fewer parameters. Pretraining improves macro-F1 by 3-4 points, while the gap between the best pretrained CNN and transformer is below 0.5 points. Architectural choices nevertheless matter: ViT-S/8 outperforms ViT-S/16 and matches the four-times-larger ViT-B/16 at a quarter of the parameters, showing that finer tokenisation benefits small cell crops. Conversely, layer-wise learning-rate decay, central to the EVA-02 fine-tuning recipe, degrades performance, highlighting that transfer heuristics from natural-image recognition may not generalise to microscopy. Finally, knowledge distillation substantially improves the deployment frontier: compact EfficientNet-B0 students distilled from teacher councils outperform every individually trained backbone, including the EfficientNet-B5 and EVA-02 teachers. Overall, our results show that rigorous control of pretraining and evaluation is essential for interpreting biomedical architecture benchmarks, while distillation may be a more effective route to practical single-cell classification than architecture choice alone.

1. Introduction

The introduction argues that conflicting CNN–ViT findings in cell microscopy may reflect differences in pretraining and evaluation, not stable architecture-family effects. It frames a controlled LIVECell benchmark to test the roles of pretraining, tokenisation, fine-tuning, and distillation.

1. Introduction

  • Recent cell-imaging comparisons report divergent CNN–ViT rankings, and the introduction identifies unmatched or unclear pretrained initialisation as a potential confound in those comparisons.The study uses LIVECell to enable comparison with prior architecture evaluations on the same dataset (Fiore et al., 2025).Source: §1 · p. 4
  • Label-free single-cell classification could support reproducible high-throughput analysis without costly labels or reagents, but morphology cues are subtle and data are imbalanced across labs and modalities.The introduction also notes domain shift across instruments, acquisition settings, and laboratories (Xing et al., 2017).Source: §1 · pp. 4–5
  • CNN locality biases may help in data-limited settings, whereas ViTs model patch interactions and may benefit from pretraining but require careful training to realize their potential.CNNs build hierarchical features from local to more global patterns; ViT attention can combine local and long-range relationships (Dosovitskiy et al., 2021).Source: §1 · p. 5
  • ViTs often underperform CNNs when trained from scratch on limited data, while large-scale pretraining can substantially improve downstream performance, including in microscopy.The introduction also notes distillation, augmentation, and strong regularisation as data-efficient strategies for ViTs (Touvron et al., 2021; Steiner et al., 2021).Source: §1 · p. 5
  • The study tests whether CNN advantages persist under matched-pretraining, source-image-disjoint LIVECell evaluation, and examines pretraining, tokenisation, fine-tuning, and distillation as performance factors.It compares EfficientNet-B0/B5, ViT-B/16, DINO-pretrained ViT-S/16 and ViT-S/8, and EVA-02, including EVA-02 layer-wise learning-rate decay and offline distillation into EfficientNet-B0.Source: §1 · p. 6

2. Results

On source-image-disjoint LIVECell splits, the strongest pretrained CNN and transformer backbones performed similarly, while pretraining and council distillation produced substantial gains. Finer tokenisation improved prediction at higher computational cost, and strong layer-wise learning-rate decay reduced performance.

2.1 Main Benchmark: Pretrained Models

  • EfficientNet-B5 led the pretrained benchmark at 91.35 ± 0.18% accuracy and 89.3 ± 0.2% macro-F1, while ViT-S/8 was close at 91.27 ± 0.16% and 89.18 ± 0.18%.ViT-S/8 exceeded ViT-S/16 by 0.9 percentage points in accuracy and 1.3 points in macro-F1, and exceeded ViT-B/16 by 0.8 and 1.0 points, respectively, using about one quarter of its parameters; the patch-8 model had comparable computational cost to ViT-B/16.Source: §2.1 · pp. 6–7, Table 1 · p. 7, Figure 1 · p. 8

2.2 Pretraining versus random initialisation

  • Pretraining raised EfficientNet-B5 accuracy by 3.4 percentage points and macro-F1 by 4.3 points, and raised ViT-B/16 accuracy by 2.7 points and macro-F1 by 3.4 points.The 90.00% accuracy of pretrained EfficientNet-B0 also exceeded the 87.99% of scratch-trained EfficientNet-B5.Source: §2.2 · p. 8, Table 2 · p. 9

2.3 Layer-wise learning-rate decay

  • LLRD at factor 0.70 reduced EVA-02 accuracy by 0.5 percentage points, macro-F1 by 0.5 points, and balanced accuracy by 0.7 points versus uniform fine-tuning.The accuracy reduction was significant (p = 0.032), whereas milder decay at factor 0.85 produced results similar to uniform fine-tuning.Source: §2.3 · p. 9

2.4 Knowledge Distillation

  • 92.40 ± 0.06% accuracy and 90.62 ± 0.09% macro-F1 made the EfficientNet-B0 student distilled from an EfficientNet-B5×3 council the numerically strongest, surpassing every individually trained backbone.Compared with the pretrained EfficientNet-B0 baseline, it gained 2.4 percentage points in accuracy and 2.9 points in macro-F1.Source: §2.4 · p. 10, Table 3 · p. 11

2.5 Computational complexity and inference efficiency

  • At batch size 64, ViT-S/8 processed approximately 2,150 images/s versus approximately 7,380 for ViT-S/16, while the distilled EfficientNet-B0 retained approximately 5,960 images/s and gained 2.9 macro-F1 points at unchanged inference cost.Reducing ViT patch size from 16 × 16 to 8 × 8 increased computation from 4.24 to 16.73 GMACs and peak inference memory from 317.7 to 760.7 MB.Source: §2.5 · p. 12, Table 6 · p. 13, Figure 2 · p. 13

3. Discussion

The discussion finds that pretraining, fine-tuning, tokenisation, and distillation shape performance and deployment trade-offs, making architecture-family rankings fragile when experimental regimes are not controlled. It also cautions that the conclusions are limited by dataset and hardware scope, with robustness under distribution shift still uncharacterised.

3.1 Pretraining and architecture-family comparisons

  • 91.35 ± 0.18% accuracy and 89.3 ± 0.2% macro-F1 for EfficientNet-B5 versus 91.27 ± 0.16% accuracy and 89.18 ± 0.18% macro-F1 for ViT-S/8 showed no significant pretrained CNN advantage.Pretraining improved EfficientNet-B5 from 85.1 to 89.3 macro-F1 and ViT-B/16 from 84.8 to 88.2; pretrained EfficientNet-B0 also outperformed scratch-trained EfficientNet-B5.Source: §3.1 · pp. 14–15, §3.2 · p. 15

3.2 Domain shift and tokenisation

  • The EVA-02 LLRD factor of 0.7 reduced accuracy from 91% to 90.5% and macro-F1 from 88.9 to 88.4, suggesting the recipe constrained adaptation to phase-contrast statistics.This tests the original EVA-02 decay recipe and does not establish that all layer-wise learning-rate decay is harmful; milder decay was near-neutral.Source: §3.2 · p. 16, §3.5 · p. 19
  • 0.9 percentage points in accuracy and 1.3 points in macro-F1 separated ViT-S/8 from ViT-S/16, while ViT-S/8 also exceeded ViT-B/16 by 0.8 and 1.0 points, respectively.The latter comparison used about one quarter as many parameters; finer tokenisation may preserve local morphological cues in small crops, though patch size was not isolated from sequence length and compute.Source: §3.1 · p. 14, §3.2 · pp. 16–17

3.3 Knowledge distillation

  • 92.40 ± 0.06% accuracy and 90.62 ± 0.09% macro-F1 made the EfficientNet-B5 council at T = 4 the numerically strongest distillation configuration, exceeding every individual backbone.Its margins over the mixed council at T = 4 were only 0.04 and 0.08 percentage points, respectively; the distilled EfficientNet-B0 retains one-pass inference cost while recovering 85–89% of direct councils’ improvement over the undistilled baseline.Source: §3.3 · pp. 17–18

3.4 Efficiency trade-offs

  • 7381 images/s and 317.7 MB made ViT-S/16 the fastest and lowest-memory evaluated model on the A100, while EfficientNet-B0 had only 4 million parameters and 0.38 GMACs.Reducing ViT patch size from 16×16 to 8×8 raised compute from 4.2 to 16.7 GMACs and memory from 317.7 to 760.7 MB, while throughput fell from 7381 to 2146 images/s.Source: §3.4 · pp. 18–19

3.5 Limitations and future work

  • The benchmark covers controlled, cropped LIVECell phase-contrast classification, leaving cross-instrument, cross-laboratory, cross-modality, and broader deployment robustness uncharacterised.Efficiency was measured on A100 GPUs under specific software and batching conditions; future evaluation should profile predictive performance, throughput, latency, memory, and robustness in realistic laboratory pipelines.Source: §3.5 · pp. 19–20

4. Methods:

The methods convert LIVECell instance annotations into single-cell crops and use source-image-disjoint splits to evaluate matched CNN and transformer models. The protocols separately examine pretraining, architecture, distillation, and deployment cost under controlled training and evaluation conditions.

4.1 Dataset Preparation

  • Parent microscopy images were split 70/15/15 within each cell line before their crops were assigned, preventing source-image overlap across train, validation, and test sets.The unbalanced class distribution was retained; crops were resized to 224×224 and trained with horizontal flips and ±5° rotations, while validation and test used deterministic resizing and normalization. Augmentation was restrained to avoid architecture-specific advantages.Source: §4.1 · pp. 20–21

4.2 Neural Architectures

  • The model set comprised EfficientNet-B0 and B5, ViT-B/16, ViT-S/8 and S/16, and pretrained EVA-02, chosen to test CNN scale, tokenisation, and transformer-family effects.EfficientNet-B0 also served as the distillation student; ViT-S/8 and S/16 tested finer patch tokenisation on small cell crops.Source: §4.2 · p. 21

4.3 Training Protocol

  • Training used AdamW, a 1×10^-4 learning rate, effective batch size 128, and three controlled seeds, with early stopping and longer budgets for randomly initialized models.Pretrained models trained up to 50 epochs with patience 5; scratch models up to 200 epochs with patience 15. EVA-02 also received a layer-wise learning-rate decay factor of 0.70 as a recipe-transfer test.Source: §4.3 · p. 22

4.4 Matched pretraining comparison

  • EfficientNet-B5 and ViT-B/16 were each tested with pretrained and random initialization under matched splits, augmentation, optimization, batch size, and evaluation.This within-architecture comparison was designed to separate initialization effects from architecture-family effects.Source: §4.4 · p. 22

4.5 Knowledge Distillation

  • Distillation trained pretrained EfficientNet-B0 students with hard-label cross-entropy and temperature-scaled teacher targets, using single teachers or averaged-logit councils.The soft-target loss received 80% weight (α = 0.2), with temperature T ∈ {3, 4}; council comparisons held the student protocol fixed and used source-image-disjoint splits.Source: §4.5 · pp. 22–23

4.6 Evaluation Metrics

  • Evaluation measured test accuracy, macro-F1, balanced accuracy, calibration error, GMACs, inference time, and peak inference memory.Compute profiling used synthetic 224×224 inputs and a single A100SXM GPU; accuracy-compute trade-offs plotted balanced accuracy and macro-F1 against GMACs, counting distilled models at student inference cost.Source: §4.6 · p. 23

Related papers

Loading 2609.09863v1…