PaperlayerView exact revision on arXiv

Source-linked AI summary

Pretraining and Distillation Matter More Than Architecture Family for Label-Free Single-Cell Classification

Philip Graemer, Giuseppe Di Caprio

arXiv:2609.09863v1cs.CV

TL;DR

The paper asks why microscopy benchmarks disagree about CNN and transformer performance for label-free single-cell classification. It uses a controlled LIVECell benchmark with disjoint source-image splits and matched protocols to separate architecture, pretraining, tokenisation, fine-tuning, and distillation effects. The results attribute the apparent CNN advantage largely to pretraining and show that council-distilled compact students can outperform individually trained backbones.

  • Problem

    Microscopy benchmarks provide conflicting evidence about whether CNNs or transformers offer the best accuracy-efficiency trade-off for label-free single-cell classification.

  • Method

    The study compares EfficientNet, ViT, and EVA-02 models on source-image-disjoint LIVECell splits using matched optimisation, augmentation, evaluation, and pretrained or random initialisation conditions.

  • Results

    Pretraining produced large gains in both families, while EfficientNet-B5 and leading transformer models showed no statistically significant separation; council distillation produced the largest improvements.

  • Takeaways & Limitations

    Pretraining and evaluation control are first-order variables in biomedical architecture benchmarks, while council distillation offers a practical route to compact high-performing classifiers.

  • Takeaways & Limitations

    The evaluation uses one controlled dataset of clean, cropped LIVECell images, so performance under cross-instrument, cross-laboratory, or cross-modality shift remains uncharacterised.

Abstract

from arXiv · show

Choosing a deep learning architecture for label-free single-cell classification remains an open question, with microscopy benchmarks reporting conflicting conclusions about CNNs versus transformers. We present a controlled benchmark on LIVECell phase-contrast microscopy data using source-image-disjoint train/validation/test splits to prevent parent-image leakage and matched optimisation, augmentation, and evaluation protocols across EfficientNet, Vision Transformer (ViT), and EVA-02 models. This allows the effects of architecture, pretraining, fine-tuning, tokenisation, and distillation to be disentangled. We find that the previously reported CNN advantage is largely explained by pretraining rather than architecture: the smallest pretrained model outperforms the strongest model trained from scratch despite far fewer parameters. Pretraining improves macro-F1 by 3-4 points, while the gap between the best pretrained CNN and transformer is below 0.5 points. Architectural choices nevertheless matter: ViT-S/8 outperforms ViT-S/16 and matches the four-times-larger ViT-B/16 at a quarter of the parameters, showing that finer tokenisation benefits small cell crops. Conversely, layer-wise learning-rate decay, central to the EVA-02 fine-tuning recipe, degrades performance, highlighting that transfer heuristics from natural-image recognition may not generalise to microscopy. Finally, knowledge distillation substantially improves the deployment frontier: compact EfficientNet-B0 students distilled from teacher councils outperform every individually trained backbone, including the EfficientNet-B5 and EVA-02 teachers. Overall, our results show that rigorous control of pretraining and evaluation is essential for interpreting biomedical architecture benchmarks, while distillation may be a more effective route to practical single-cell classification than architecture choice alone.

1. Introduction

The paper addresses conflicting CNN-versus-transformer findings in microscopy by benchmarking architectures under matched training conditions. It focuses on separating architecture effects from pretraining, fine-tuning, tokenisation, and distillation.

  • CNN-versus-transformer comparisons in cell imaging report divergent conclusions across microscopy benchmarks.
  • The study argues that unmatched or poorly reported pretrained initialisation can confound comparisons between architecture families.
  • Microscopy classification remains difficult because discriminative cues can be subtle and local, labels costly and noisy, and deployment affected by domain shift.
  • The benchmark evaluates whether the reported CNN advantage persists under source-image-disjoint, matched-pretraining conditions.
  • It also tests whether pretraining, transformer tokenisation, EVA-02 fine-tuning, and knowledge distillation explain performance and deployment differences.

2. Results

Controlled LIVECell experiments show that pretraining and training choices strongly affect classification, while model architecture still matters through tokenisation and distillation improves deployment efficiency.

  • Experimental setup: Source-image-disjoint LIVECell splits and matched optimisation, augmentation, and evaluation protocols enabled controlled comparisons across six architectures.The benchmark included EfficientNet-B0/B5, ViT-B/16, ViT-S/16, ViT-S/8, and EVA-02.
  • Architectural factors: 0.9 percentage points in accuracy and 1.3 points in macro-F1 separated ViT-S/8 from ViT-S/16 under matched width and DINO pretraining.The comparison principally changes patch size, providing evidence that finer tokenisation benefits small single-cell crops.
  • Pretraining versus random initialisation: 3.4 percentage points in accuracy and 4.3 points in macro-F1 were gained by pretrained EfficientNet-B5 over random initialisation.Pretrained EfficientNet-B5 improved from 87.99 ± 0.10% accuracy and 85.07 ± 0.03% macro-F1 to 91.35 ± 0.18% and 89.3 ± 0.2%.
  • Fine-tuning strategy: 0.5 percentage points lower accuracy and macro-F1 followed EVA-02 fine-tuning with layer-wise learning-rate decay factor 0.70 versus uniform fine-tuning.The accuracy reduction was statistically significant, while macro-F1 and balanced-accuracy reductions were directionally consistent but not conventionally significant.
  • Knowledge distillation: 92.40 ± 0.06% accuracy and 90.62 ± 0.09% macro-F1 were achieved by the EfficientNet-B5×3 council at T=4.Council distillation used averaged teacher logits to train compact EfficientNet-B0 students.
  • Computational efficiency: 2.9 points lifted the attainable macro-F1 frontier when the council-distilled EfficientNet-B0 retained approximately 5960 images/s at unchanged inference cost.The student exceeded the macro-F1 of every individually trained model while using student inference cost only.

3. Discussion

Under controlled conditions, the reported CNN advantage largely disappears: pretraining is a first-order determinant of performance, and architecture-family rankings depend on the training regime. The benchmark therefore treats pretraining, split protocol, tokenisation, and fine-tuning recipe as integral experimental variables.

  • Pretraining and architecture-family comparisons: 3.4 percentage points: pretrained EfficientNet-B5 improved from 88% accuracy with random initialisation to 91.4% with pretrained weights.Its macro-F1 increased from 85.1 to 89.3.
  • Pretraining and architecture-family comparisons: 2.7 points: pretrained ViT-B/16 improved from 87.8% accuracy with random initialisation to 90.5% with pretrained weights.Its macro-F1 increased from 84.8 to 88.2.
  • Pretraining and architecture-family comparisons: Pretrained EfficientNet-B0 outperformed scratch-trained EfficientNet-B5 despite being the smaller and weaker CNN.This shows that initialisation can dominate model scale in the reported comparison.
  • Pretraining and architecture-family comparisons: The large previously reported CNN advantage does not survive controlled experimental conditions.The corrected benchmark attributes the earlier gap to an interpretation that did not isolate architecture family from initialisation regime.
  • Pretraining and architecture-family comparisons: Architecture-family rankings are fragile unless pretraining, split protocol, tokenisation, and fine-tuning recipe are controlled and explicitly reported.The discussion presents pretraining regime as a primary experimental variable rather than a minor implementation detail.

3.2 Domain shift and tokenisation

The strongest pretrained CNN and transformer perform similarly, while within the ViT family finer tokenisation benefits small single-cell crops. However, the patch-size interpretation remains limited because the comparison does not isolate locality from sequence length and computational cost.

  • Domain shift and tokenisation: 91.35 ± 0.18% accuracy and 89.3 ± 0.2% macro-F1: EfficientNet-B5 numerically led ViT-S/8, but neither comparison was statistically significant.For ViT-S/8, the corresponding values were 91.27 ± 0.16% accuracy and 89.18 ± 0.18% macro-F1; p = 0.069 and p = 0.076, respectively.
  • Domain shift and tokenisation: The results provide no evidence of residual CNN superiority under pretrained initialisation.The discussion links this comparison to the mismatch between natural-image representations and phase-contrast microscopy statistics.
  • Domain shift and tokenisation: 0.8 percentage points in accuracy and 1 point in macro-F1: ViT-S/8 outperformed the larger ViT-B/16.The result suggests that finer patch tokenisation can preserve fine-grained local morphology in small cell crops more effectively than increasing model scale.
  • Domain shift and tokenisation: At 224 × 224 resolution, ViT-B/16 uses a 14 × 14 grid of 196 tokens, making its patches coarser than ViT-S/8 for the same crop size.The discussion identifies membrane texture, nuclear granularity, cytoplasmic organisation, boundary irregularity, and halo structure as potentially local cues.
  • Domain shift and tokenisation: The patch-size comparison does not isolate locality from sequence length and computational cost, so the proposed mechanism remains an interpretation rather than a demonstrated mechanism.Because crops are upsampled to 224 × 224, patch size determines granularity relative to cellular structure rather than directly determining pixel-level resolution.

3.3 Knowledge distillation

Knowledge distillation changes the deployment trade-off: compact EfficientNet-B0 students can outperform individually trained backbones while retaining compact-model inference cost. Direct councils perform somewhat better, but distilled students recover most of the council benefit at substantially lower cost.

  • Knowledge distillation: All council configurations achieved higher mean accuracy and macro-F1 than any individual backbone while retaining EfficientNet-B0 inference cost.The strongest student was distilled from the EN-B5×3 council.
  • Knowledge distillation: 91.5% accuracy and 89.6 macro-F1: EfficientNet-B0 reached these values after EfficientNet-B5 teacher distillation at T = 4.The undistilled EfficientNet-B0 baseline reached 90% accuracy and 87.7 macro-F1.
  • Knowledge distillation: 0.30–0.39 percentage points in accuracy and 0.38–0.46 points in macro-F1: direct councils outperformed their corresponding distilled students.Despite this gap, distilled students retained approximately 85–89% of the direct councils’ improvement over undistilled EfficientNet-B0.
  • Knowledge distillation: Council distillation recovered most of direct ensembling’s benefit using only a single EfficientNet-B0 forward pass at inference.Direct ensembling maximised predictive performance, whereas council distillation reduced inference cost substantially.
  • Knowledge distillation: On A100 batch inference, ViT-S/16 reached 7381 images/s, whereas EfficientNet-B0 reached 5956 images/s despite the latter’s smaller parameter and GMAC counts.The efficiency ranking therefore depends on hardware and batch configuration, not parameter count or GMACs alone.

3.5 Limitations and future work

The benchmark is limited to controlled LIVECell crops from a single public dataset, so robustness under broader distribution shift and realistic laboratory context remains uncharacterised. Future evaluations should include end-to-end deployment conditions and contextual information beyond isolated single-cell crops.

  • Limitations and future work: The conclusions apply most directly to controlled cropped phase-contrast classification from one public dataset.LIVECell crops are cleanly segmented, background-reduced, and morphologically consistent compared with typical laboratory data.
  • Limitations and future work: Performance under cross-instrument, cross-laboratory, or cross-modality conditions remains uncharacterised.The relative architecture ordering may shift under stronger distribution shift.
  • Limitations and future work: Single-cell crops in isolation suppress field-level acquisition and contextual cues that may be useful in deployment.Real laboratory pipelines may require segmentation, tracking, quality control, and information from neighbouring cells, local confluence, and colony organisation.
  • Limitations and future work: Training memory and end-to-end pipeline latency were not systematically profiled.The unprofiled stages include image loading, preprocessing, and host-to-device transfer.
  • Limitations and future work: Future work should evaluate accuracy, throughput, latency, memory, and robustness under realistic laboratory deployment conditions.The discussion also motivates combining a target-cell crop with a representation of its neighbourhood or full field of view.

4.1 Dataset Preparation

The study uses LIVECell, a large-scale label-free phase-contrast microscopy dataset, and restrained augmentation to preserve comparability across models.

  • Dataset: LIVECell contains 5,239 images with approximately 1.6 million annotated cells across eight distinct cell lines.The cell lines are A172, BT-474, BV-2, Huh7, MCF7, SH-SY5Y, SkBr3, and SK-OV-3.
  • Dataset: The dataset provides label-free phase-contrast microscopy images with diverse cell-line morphologies.It is described as highly normalised with little background.
  • Augmentation: Restrained augmentation was chosen to preserve comparability with the prior benchmark and avoid architecture-specific advantages on small phase-contrast cell crops.The study did not test whether stronger augmentation would improve this modality.
  • Dataset: Table 7 reports instances per class and each class's percentage of the total dataset.

4.2 Neural Architectures

The benchmark compares convolutional and transformer-family models selected to test compactness, scale, tokenisation, and pretraining effects.

  • Model families: EfficientNet-B0 serves as the compact convolutional baseline and distillation student, while EfficientNet-B5 represents a stronger mid-scale CNN.
  • Model families: ViT-B/16 is the standard Vision Transformer baseline, while ViT-S/8 and ViT-S/16 test finer patch tokenisation for small cell crops.
  • Model families: EVA-02 is included as a modern pretrained Vision Transformer family model.
  • Model families: Table 8 summarizes the neural architectures and pretraining regimes used in the benchmark.

4.3 Training Protocol

Models were trained under a controlled PyTorch and timm protocol using matched optimisation settings, restrained augmentation, and memory-efficient training procedures.

  • Optimisation: Training used AdamW with learning rate 1×10^-4 and effective batch size 128 via gradient accumulation.
  • Training procedure: The protocol used restrained geometric augmentation, mixed-precision bfloat16 where supported, and activation checkpointing for memory-intensive transformers.
  • Training procedure: Pretrained models were fine-tuned end-to-end for up to 50 epochs with early-stopping patience of 5.

4.4 Matched pretraining comparison

The matched comparison isolates the contribution of pretraining from architecture family by evaluating EfficientNet-B5 and ViT-B/16 under pretrained and random initialisation.

  • Comparison design: EfficientNet-B5 and ViT-B/16 were each evaluated with pretrained and random initialisation.
  • Comparison design: The comparison matched data splits, augmentation, optimiser, effective batch size, and evaluation protocol across initialisation conditions.
  • Comparison design: This within-architecture design quantifies how much the CNN–ViT ranking is attributable to initialisation rather than architecture alone.

4.5 Knowledge Distillation

The distillation setup combines hard ground-truth supervision with temperature-scaled teacher predictions, and compares individual teachers with councils under a fixed student protocol.

  • Objective: The total distillation loss combines cross-entropy on hard labels with temperature-scaled KL divergence on soft teacher targets.The soft-target term receives 80% of the loss weight, with α = 0.2 and T ∈ {3, 4}.
  • Teacher councils: Council distillation averages teacher logits sample-wise before converting them into a temperature-scaled soft-target distribution.
  • Experimental design: The experiments compare single-teacher pairings with homogeneous and mixed three-teacher councils while holding the EfficientNet-B0 student architecture fixed.Council compositions include EVA-02×3, 2×EVA-02 plus EfficientNet-B5, and EfficientNet-B5×3.
  • Experimental design: All distillation runs use the same student initialisation, optimisation, augmentation, data-fraction, validation, and evaluation protocols.This design isolates differences attributable to teacher identity or council composition.

4.6 Evaluation Metrics

Evaluation combines classification metrics with standardized computational profiling and visualizes accuracy–compute trade-offs, while associated data and code are made available for reproducibility.

  • Classification metrics: Classification performance is assessed using test accuracy, macro-F1, balanced accuracy, and expected calibration error.
  • Compute profiling: Model complexity is measured uniformly with PyTorch FLOP counting on a synthetic 1×3×224×224 input, reporting GMACs as forward FLOPs divided by two.Inference profiling also records peak CUDA memory and uses fixed inputs, inference mode, bf16 autocast, and a single A100SXM GPU.
  • Accuracy–compute trade-offs: Accuracy–compute trade-offs are plotted using balanced accuracy and macro-F1 against GMACs, with distilled models evaluated at student inference cost.
  • Reproducibility: The study provides public LIVECell-derived crops, source-image-disjoint split assignments, training and profiling code, and configuration files.
Loading 2609.09863v1…