Source-linked AI summary
Picking the Right Image to Classify: Reliable-Input Selection in Teledermatology
Fabian Gröger, Marco Weishaupt, Philippe Gottfrois, Simone Lionetti, Linda Wermelinger, Nipun Ranasekara, Ludovic Amruthalingam, Alexander A. Navarini, Marc Pouly
TL;DR
Teledermatology models can misclassify ordinary images captured under acquisition conditions unlike their training data, motivating selection of the most reliable image when multiple case images exist. This paper benchmarks training-data-free and small-reference-set selectors across dermatology datasets and frozen backbones, finding that none recovers more than a small fraction of the oracle gain, leaving reliable-input selection unsolved.
Problem
Teledermatology acquisition shifts can make ordinary clinical images difficult for dermatology models, creating a need to select the most reliably classified image among multiple case images.
Method
The study benchmarks embedding norm, neighborhood consensus, perturbation stability, confidence, and reference-set selectors across six datasets and nine frozen backbones.
Results
The best selector, a confidence–Mahalanobis fusion, improves weighted F1 by +4.4 on average but recovers at most about a quarter of the oracle gap.
Takeaways & Limitations
Reliable-input selection remains unsolved, because neither training-data-free selectors nor a small labeled reference set substantially narrows the oracle gap.
Takeaways & Limitations
Current frozen encoders do not expose a sufficiently reliable signal of their own competence for successful image selection.
Abstract
from arXiv · showhide
Dermatology models face distribution shifts in teledermatology settings, where submitted images differ from the training data in lighting, angle, distance, focus, and framing. These test-time images are ordinary clinical photographs, but some fall outside the model's training conditions, leading the model to often misclassify them due to shifts in acquisition between training and deployment. When multiple images of the same case exist (several photos of one patient or lesion), a natural way to improve accuracy is therefore to select the image the model is most likely to classify correctly. We call this task reliable-input selection. An oracle that, for each case, selects a correctly classified image when one exists raises weighted F1 by about 20 percentage points on average across six dermatology datasets and nine frozen backbones. This oracle is an upper bound that sees the labels, whereas a selector must choose blindly. Capturing this gain in practice is hard. A selector that needs no pretraining data applies to any frozen model, including those whose data is not public. It must judge reliability from quantities the model exposes at inference: its embeddings, their norms, and its confidence. We benchmark four such training-data-free selectors: the embedding norm, the neighborhood consensus among a case's images, the stability of the prediction under small perturbations, and the model's own confidence. No training-data-free selector substantially narrows this oracle gap. The best of them is the model's own confidence, but it recovers only a small part of the gap on the clinical datasets. A small labeled reference set does not help either: the best selector overall, a fusion of confidence and Mahalanobis distance, still leaves most of the gap. To our knowledge, this is the first study to introduce and benchmark reliable input selection, a clinically important, unsolved task.
1 Introduction
Teledermatology acquisition shifts can make ordinary clinical images silently and confidently misclassified. Reliable-input selection chooses, among multiple case images, the one a frozen model is most likely to classify correctly, but existing selectors recover little of the available gain.
- Motivation: Acquisition shifts in angle, distance, and lighting cause dermatology models to misclassify ordinary clinical photographs, often with apparent confidence.These images are not anomalies, so an out-of-distribution detector would not flag them.
- Task definition: Reliable-input selection chooses, among multiple images from one case, the image a deployed model is most likely to classify correctly.Studied cases include repeated lesion captures, multiple lesion viewpoints or modalities, and patient photographs covering different body sites.
- Selector results: Four training-data-free selectors and reference-set selectors recover little of the oracle gap across six datasets and nine backbones.Training-data-free methods operate on frozen-model quantities without requiring pretraining data, while reference-set methods may use a small labeled set.
- Oracle opportunity: About 20 percentage points of weighted F1 is the average oracle gain across six datasets, establishing a large upper-bound gap for selection.The oracle sees labels and selects a correctly classified image when one exists.
- Selector results: At most about a quarter of the gap is recovered by the best selector, which fuses confidence with a class-conditional Mahalanobis distance.The model’s own confidence helps on multi-image clinical datasets, but the overall results remain largely negative.
2 Related Work
Prior work shows that acquisition shifts degrade model accuracy and that confidence-based rejection addresses reliability differently from selecting among valid images. Dermatology foundation models, coverage gaps, and redundant video frames further motivate deployment-time image selection.
- Generalization under acquisition shift: Acquisition shifts degrade accuracy, and robustness to synthetic perturbations does not reliably transfer to natural shifts.The cited literature identifies larger, more diverse training data as the main known remedy.
- Confidence, OOD detection, and sample rejection: Selective prediction typically rejects unreliable or out-of-distribution inputs, whereas reliable-input selection chooses among several valid images without abstaining.The distinction matters because unfamiliar acquisition conditions are valid inputs rather than anomalies.
- Foundation models and video in dermatology: Dermatology foundation models offer strong embeddings with little labeled data, but coverage gaps and redundant video frames make deployment-time reliability important.Selecting reliable frames is presented as a direct companion to video-based systems that exploit frame redundancy.
3 Methods
The method selects one image per case using frozen-backbone features and evaluates downstream diagnostic performance with weighted F1. It compares training-data-free and small-reference-set selectors across diverse frozen backbones, probes, datasets, and acquisition settings.
- Case representation and evaluation: Each case is represented by multiple images, a frozen backbone embeds them, and a selection rule chooses one image whose prediction determines weighted F1.Weighted F1 is used because the diagnostic classes are imbalanced.
- Backbones and probes: The evaluation spans nine unfine-tuned backbones covering dermatology, contrastive, supervised, masked-autoencoding, and self-distillation pretraining paradigms.Four DINOv2 variants are included: plain, register, patch-mean, and register-pluspatch-mean.
- Selection rules: Selectors score each image and choose the per-case arg max, with random selection as the universal baseline and majority and soft voting as non-selection comparisons.Majority vote uses the most frequent prediction, while soft vote uses the argmax of mean predicted probability.
- Training-data-free selectors: Training-data-free selectors use only frozen embeddings and the probe, including embedding-norm typicality, neighborhood consensus, and perturbation stability.Neighborhood consensus favors the most central embedding, while perturbation stability favors predictions that change least under small Gaussian embedding perturbations.
- Reference-set selectors: Reference-set selectors use a small labeled set to estimate class-conditional typicality with Mahalanobis distance or fuse that score with classifier confidence.These selectors are fitted without access to the model’s pretraining data.
- Backbones and probes: Frozen representations are evaluated with logistic and kNN probes using k=5 and cosine distance, stratified 60/40 case splits, and averages over 10 seeds.Each probe is trained on one acquisition type and applied to every image in held-out cases.
4 Experiments and Results
A label-aware oracle improves weighted F1 substantially, but practical selectors recover only a small fraction of this opportunity. The main limitation is weak reliability signal: confidence is the only clearly useful training-data-free score, and its gains remain modest across datasets and regimes.
- Oracle gap: About 20 percentage points: the oracle’s average F1 gain over random selection far exceeds the 2–8-point advantage of the fixed best acquisition.The fixed acquisition is defined only on aligned datasets and is used as a target rather than a deployable baseline.
- Selector performance: +3.7 on average: model confidence is the strongest training-data-free selector, while confidence fused with class-conditional Mahalanobis distance reaches +4.4 overall.Both recover at most about a quarter of the oracle gap, leaving reliable-input selection unsolved.
- Why selection is hard: 31% to 49%: mixed cases containing both correct and incorrect images are common, so limited rescue opportunity is not the main bottleneck.Cases with no correctly classified image cannot be rescued by selection, whereas selection is irrelevant for all-correct cases.
- Why selection is hard: AUROC 0.58 to 0.70: confidence is the only training-data-free score clearly above chance for identifying correct images in mixed cases.Embedding norm is at chance, neighborhood consensus is barely above chance, and perturbation stability lies between them.
- Robustness across settings: 36%, 33%, 31%: on SCIN, the oracle’s correct selections spread nearly evenly across the three viewpoints, showing that no single fixed viewpoint consistently suffices.The best image changes from case to case.
- Robustness across settings: Under 2 points: oracle-gap variation across nine backbones is far smaller than variation across datasets, and the gap remains large across all regimes at +14 to +24 points.Confidence recovers most in the per-patient regime (+5) and less elsewhere (+3).
5 Discussion and Conclusion
Reliable input selection is clinically relevant because choosing among a case’s images adds about 20 percentage points of weighted F1 on average, yet remains largely unsolved without pretraining data. Current frozen encoders do not expose a sufficiently reliable competence signal, with same-case images receiving different predictions 34% to 69% of the time.
- Clinical relevance: About 20 percentage points of weighted F1 are added on average by choosing the input image, especially in clinical teledermatology collections.The result spans six datasets.
- Unresolved task: No training-data-free selector recovers more than a small fraction of the reliable-input gap across nine backbones, and a small labeled reference set still leaves most of it.These findings indicate that reliable input selection remains unsolved without the model’s pretraining data.
- Encoder limitations: Current frozen encoders do not provide a reliable signal of their own competence, limiting blind selection of correctly classified images.The authors identify heavy self-supervised-pretraining augmentation as a likely obstacle because it encourages, but does not achieve, invariance to acquisition changes.
- Encoder limitations: 34% to 69% of same-case image pairs receive different predictions, averaging 50% disagreement.This prediction instability shows that images of the same case can produce substantially different model outputs.