Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis

1. Introduction

• Labels are often scarce because annotation is costly or time-consuming, may not be needed in clinical practice, and collected data is not always shared with other researchers.The paper cites Litjens et al. (2017), Weese and Lorenz (2016), and de Bruijne (2016) on the lack of labeled medical images.Source: §1 · p. 1
• Semi-supervised, multiple instance, and transfer learning use other available data or labels, but their related goals have had little interaction.The survey frames this limited interaction as a missed opportunity and aims to describe connections, gaps, and future research opportunities.Source: §1 · p. 1

1.1. Selection of papers

• The survey selects medical imaging papers from searches for three learning-scenario terms, includes papers available online before 2018, and emphasizes scenarios and assumptions over classifiers or exhaustive paper summaries.Similar papers were represented by the latest paper; later suggestions were included if they met the criteria.Source: §1.1 · p. 2
• The survey focuses on classification for diagnosis, detection, or segmentation and excludes detailed coverage of approaches requiring annotator interaction, such as active learning or crowdsourcing.Its focus is on approaches usable when additional labels cannot be acquired.Source: §1.1 · p. 2

2. Overview of techniques

• In supervised learning, labeled training patches are used to classify patches in previously unseen scans from the same domain.The example labels patches as emphysema or normal tissue.Source: §2 · p. 2
• SSL adds unlabeled training data to improve classifier predictions, such as unlabeled chest CT patches alongside manually labeled patches.The test data are part of the same learning scenario described above.Source: §2 · p. 2
• MIL trains on labeled bags with unknown instance labels related to each bag label, then can classify test bags, instances, or both.For example, a radiologist may label an entire scan for emphysema without marking its locations; instance classification can localize emphysema.Source: §2 · p. 2
• TL addresses source and target data from different domains, training with source data and potentially unlabeled test data or labeled target-domain data.Different scanning protocols can produce different patch appearances and distributions; TL is not limited to differences in feature distributions.Source: §2 · p. 2

3. Semi-supervised learning

• SSL methods exploit assumptions linking feature distributions to labels, but can underperform supervised learning when those assumptions fail; newer approaches instead use assumptions already present in the classifier.Common assumptions are that nearby or clustered samples share labels and that class boundaries lie in low-density regions; self-training, expectation-maximization, co-training, graph methods, and semi-supervised SVMs use these ideas, while newer methods can link parameter estimates from labeled and all samples.Source: §3 · pp. 4–5

3.1. SSL in medical imaging

• In medical imaging, partially annotated images and diagnostically ambiguous subjects can provide unlabeled samples; surveyed work uses self- or co-training for segmentation and graph or SVM regularization for segmentation and diagnosis.For example, subjects with mild cognitive impairment may be treated as unlabeled in Alzheimer’s disease versus normal-cognition classification.Source: §3.1 · p. 5

3.2. Self-training and co-training

• Self-training repeatedly classifies unlabeled samples and adds all or a selected subset to the training set; selection may use expert verification, confidence thresholds, or agreement among classifiers.Active-learning variants require expert interaction, which this survey does not address in detail; co-training examples include skull segmentation and tuberculosis-pattern classification.Source: §3.2 · p. 5
• Self-training is most common for segmentation across brain, retina, and heart applications, while surveyed diagnosis or detection examples classify chest-CT volumes or cell nuclei with sample sizes in the thousands.The survey suggests self-training is more often used where larger datasets are common, as in segmentation, than in computer-aided diagnosis.Source: §3.2 · p. 5
• Increasing sample size improves performance in several surveyed studies, while the advantage of semi-supervised methods decreases as more labeled data becomes available.This pattern is reported by Iglesias et al. (2010), Bai et al. (2017), and Gu et al. (2017).Source: §3.2 · p. 5

3.3. Graph-based methods and regularization

• Graph-based methods propagate labels across similar connected samples, but graph-cut labeling has an out-of-sample limitation; manifold regularization instead uses graph smoothness to label unseen data.Graph cuts are used for pixel or superpixel segmentation and require labeled pixels for a previously unseen image; atlas-based methods propagate labels between neighboring images. Manifold regularization is used in both segmentation and computer-aided diagnosis.Source: §3.3 · p. 7
• Semi-supervised SVMs favor hyperplanes that place unlabeled samples outside the margin, enforcing low-density separation for Alzheimer’s disease or mild cognitive impairment classification.This approach is used by Filipovych et al. (2011) and Moradi et al. (2015).Source: §3.3 · p. 7

4. Multiple instance learning

• MIL trains from labeled bags of unlabeled instances; under the standard assumption, a bag is positive if and only if it contains at least one positive instance, though collective assumptions let all instances contribute.Methods may infer instance labels and aggregate them, such as with noisy-or, or classify a bag-level feature representation directly; the latter typically cannot provide instance labels.Source: §4 · p. 7

4.1. MIL in medical imaging

• Medical-image MIL covers global detection, local detection, and false-positive reduction; in the latter, a candidate lesion is the bag and its different views are correlated instances.False-positive reduction focuses on classifying the candidate bag, so instance classification is less relevant than in global or local detection.Source: §4.1 · p. 8

4.2. Global detection

• For global detection, bag-level methods seem more successful than instance-level methods because they account for instance correlations and structure, and can even outperform fully supervised methods (Kandemir and Hamprecht, 2015; Wang et al., 2015a; Vural et al., 2006; Samsudin and Bradley, 2010).Applications include diabetic-retinopathy and histopathology cancer detection; reported datasets range from 58 bags to 100K chest x-rays.Source: §4.2 · p. 8

4.3. Global and local detection

• Global and local detection may be pursued together, as in tuberculosis detection, but local detection is often unevaluated when instance labels for validation are unavailable (Cheplygina et al., 2015a; Kandemir and Hamprecht, 2015).For tuberculosis, the goal can be both to classify an image and to highlight lesions.Source: §4.3 · p. 8

4.4. Local detection only?

• Local-detection methods also optimize global detection, so the “local detection only” category is technically empty.The authors retain the category to address the perceived distinction.Source: §4.4 · p. 10
• “Weakly supervised” often refers to MIL with bag labels, but the terms are not disjoint.The authors note that this usage is common when global detection is not addressed and could create the false impression that MIL and weak supervision are separate.Source: §4.4 · p. 10
• “Weakly supervised” also describes methods outside MIL, including few-annotation label propagation and learning from noisy labels.Donner et al. (2009) uses few annotations to initialize label propagation, classified here as SSL; Rajchl et al. (2016) uses the term for noisy labels, a variation outside the survey’s scope.Source: §4.4 · p. 10

4.5. False positive reduction with multiple viewpoints

• False positive reduction within MIL classifies a candidate tumor or lesion as a bag of multiple viewpoints, with the bag label applying to every instance.Quellec et al. (2017) describes this task; instances are viewpoints of the same candidate. As in global detection, combining instance information can let an MIL classifier outperform a supervised classifier, while identifying influential instances is less relevant because the candidate is classified as a whole.Source: §4.5 · p. 10

5. Transfer learning

• Transfer learning allows source and target problems to differ in domain, task, or both, with domain shifts potentially involving p(x), p(y), or p(y|x).Domains are defined by feature space and sample distribution, while tasks are defined by label space and the mapping from features to labels.Source: §5 · p. 11
• The survey groups transfer methods as instance transfer, which reweights source data for the target, and feature transfer, which encodes source knowledge in target representations.Source: §5 · p. 11

5.1. TL in medical imaging

• For the same images with different classification tasks, feature transfer learns representations useful across tasks, while unsupervised pretraining such as reconstruction is another option (Ruder, 2017).Shared features can increase effective sample size and/or regularize the classifier.Source: §5.1 · p. 11
• For different domains with the same task, instance transfer can weight relevant source samples or align source and target distributions before training on their combined instances.The survey groups alignment with instance transfer because both approaches reduce irrelevant samples and/or increase relevant ones.Source: §5.1 · p. 11
• When both task and domain differ, a source-task-pretrained network can support target learning through feature extraction or fine-tuning (Litjens et al., 2017).The survey notes both strategies are popular in medical image analysis.Source: §5.1 · p. 11

5.3. Different domains, same task

• Instance weighting can match source instances to the target distribution, which is optimal when marginal distributions differ but labeling functions are the same and can still help when they differ.Weights may instead use patient characteristics such as age, even when those characteristics are not classifier inputs.Source: §5.3 · p. 14
• Using an invariant representation learned from similar and dissimilar cross-scanner voxels, training on their union outperformed source-only, target-only, and original-representation union training (Kouw et al., 2017).Other alignment approaches use PCA, scanner correspondences, or Gaussian dataset representations before source-domain training.Source: §5.3 · p. 14
• Transfer across domains may be unsupervised or use a small amount of labeled target data, as illustrated by the surveyed works.Source: §5.3 · p. 14

5.4. Different task, different domains

• Deep transfer uses source-trained networks to extract off-the-shelf target features or as starting points for fine-tuning, often compared with training from scratch.Transfer from natural images, including ImageNet datasets, is common in practice.Source: §5.4 · pp. 14–15
• For tissue classification in chest CT, natural-image pretraining performed comparably to or slightly better than lung images, while brain images were less effective; Places pretraining also improved polyp recognition over ImageNet.The brain-image result was possibly due to large homogeneous regions, whereas lung CT contains more texture; Zhang et al. hypothesized that Places' greater similarity between classes helped distinguish subtle polyp differences.Source: §5.4 · p. 15
• For melanoma classification, fine-tuning outperformed off-the-shelf features, and ImageNet alone outperformed KaggleDR or their union (Menegola et al., 2017).The weaker result for the union suggests dataset size alone does not explain transfer success; for bladder cancer-treatment response, transfer and no-transfer networks showed no statistically significant differences. The review suggests a trade-off among source similarity, size, and possibly diversity.Source: §5.4 · p. 15
• For polyp classification, texture datasets performed best as pretraining sources, while increasing the number of images and classes did not always improve performance.The comparison used source datasets matched in number of classes and images.Source: §5.4 · p. 15

6.1. Trends

• Transfer learning has become the most popular surveyed scenario only in recent years, possibly because external datasets and pretrained models are readily available.Unlike SSL and MIL, TL can use a completely external dataset alongside target data.Source: §6.1 · p. 15, Figure 2 · p. 16
• Brain is the most common application, followed by histology/microscopy and abdomen; MIL is more common in histology than brain because its unordered, variable patches fit that setting.Breast, heart, and retina have relatively few papers, while around 10% address multiple applications. Brain anatomy has informative correspondences, making MIL less applicable there.Source: §6.1 · p. 16, Figure 3 · p. 16

6.2. Related learning scenarios

• Comparing related learning scenarios could reveal better matches between problem assumptions and methods, including whether MIL’s bag constraints matter relative to SSL.MIL can be viewed as a special case of SSL: instances in negative bags are labeled, while positive-bag instances are unlabeled subject to the constraint that not all can be negative.Source: §6.2 · p. 16
• The discussion links SSL, MIL, and TL and highlights positive-unlabeled learning and MIL variants as underexplored opportunities.Domain information may favor TL over SSL, and earlier multi-center SSL work may have involved unrecognized transfer settings. Same-domain bag labels can substitute for different-domain instance labels, while MIL can also relate to multi-task or self-transfer learning. Positive-unlabeled learning has only one directly addressing paper found, Zuluaga et al. (2011); batch classification and group-based learning are MIL variations whose differences may illuminate the diversity of MIL problems.Source: §6.2 · pp. 16–17

6.3. Full potential of available data

• Available labeled data may be underused: MIL studies sometimes turn regression or multi-class tasks binary or remove difficult classes; Menegola et al. (2016) found removing one disease class lowered performance, possibly because fewer samples remained.Modeling individual expert labels during training can outperform pre-averaged consensus labels (Guan et al., 2017); clinical variables may be additional outputs or auxiliary tasks, while reports provide additional information (Ruder, 2017).Source: §6.3 · p. 17

6.4. Acquiring additional labels

• Active learning can prioritize ambiguous or high-impact labels, while integrating crowdsourced labels into learning methods remains an important research direction.The surveyed active-learning methods aim to reduce labels needed for comparable or better performance; crowdsourcing studies more often assess label quality than use those labels within machine learning.Source: §6.4 · p. 17

6.5. Generalization

• Because most studies are proofs of concept on few applications, generalization is difficult, and publication bias and single-application benchmarks compound the concern.The discussion proposes varying dataset characteristics and ablating methods, addressing publication bias, and evaluating methods across datasets from different medical applications rather than only single-application challenges.Source: §6.5 · p. 18

• The survey discusses over 140 papers on “not-so-supervised” learning for medical image classification, often driven by a lack of representative annotated data.Source: §7 · p. 18
• Transfer learning has been the most popular of the reviewed approaches in recent years, alongside semi-supervised and multi-instance learning.Source: §7 · p. 18
• Although individual papers show these approaches are useful, questions remain about how best to use them; future research could examine links between learning scenarios and generalize findings across applications.Source: §7 · p. 18