PaperlayerView exact revision on arXiv

Source-linked AI summary

L2GSSL: Label-Efficient Local-to-Global Self-Supervised Learning for Volumetric Healthcare Data

Po-Cheng Cheng, Hung-Hsuan Chen

arXiv:2610.21356v1cs.CVcs.LG

TL;DR

CT archives contain far more scans than experts can label, and each volume combines local slice content with ordered volumetric context. L2GSSL separates self-supervised learning across those local and global spaces. Its reported gains are greatest with limited labels and diminish under full supervision.

  • Problem

    CT scans are abundant and high-dimensional, but expert-curated labels cover only a fraction, while useful representations must capture both local pathology and volumetric context.

    Source: §I · p. 1

  • Method

    L2GSSL separately regularizes local slice and global volume representations, using stop-gradient to prevent the global objective from updating the local encoder.

    Source: §I · p. 1, A. Hierarchical Representation Spaces · p. 3

  • Results

    L2GSSL’s largest reported gains occur with 10% and 25% of downstream labels and diminish with full supervision.

    Source: §I · p. 1

  • Takeaways & Limitations

    Slice-wise execution and phased optimization support hierarchical volumetric learning within the measured constrained GPU-memory budgets.

    Source: A. Principal Findings · p. 9, §VI · p. 10

    The evaluation is limited to retrospective CT classification and does not cover prospective deployment or other imaging modalities.

    Source: D. Limitations · pp. 9–10

Abstract

from arXiv · show

Healthcare institutions routinely accumulate high-dimensional computed tomography (CT) volumes, but only a fraction can be annotated by experts. We present L2GSSL, a local-to-global self-supervised learning framework that leverages unlabeled volumes by decomposing pretraining into intra-slice and inter-slice representation spaces. A slice encoder learns view-invariant local features, and a global encoder aggregates the resulting tokens to model volumetric context. The stop-gradient isolates the local space from the global objective, while slice-wise execution and a three-phase schedule support training under constrained GPU memory. On LIDC-IDRI lung nodule malignancy classification, L2GSSL reaches $0.9229$ AUROC with only $10\%$ of the labels and $0.9377$ with $25\%$, exceeding the volume-level VICReg by $0.0229$ and $0.0294$, respectively. With all labels, the strongest supervised baseline exceeds L2GSSL by only $0.0044$ AUROC, consistent with self-supervised pretraining providing its greatest value when annotations are limited. Without adaptation to the external LNDb cohort, a model fine-tuned on $25\%$ of LIDC-IDRI obtains $0.8138$ AUROC and $0.7589$ AUPRC, improving over the strongest evaluated self-supervised baseline by $0.1130$ and $0.0954$. Experiments across data sources, thoracic and abdominal anatomy, downstream tasks, and two backbones examine healthcare-data variety, while lesion-slice masking shows that the predictions depend on lesion-bearing content.

I. INTRODUCTION

CT archives are abundant, but expert labels and accelerator memory are limited, while scans combine slice-level pathology with ordered volumetric context. L2GSSL separates local slice and global volume objectives, and its reported gains are greatest with limited downstream labels.

I. INTRODUCTION

  • CT volumes combine fine pathological appearance, organ-scale structure, and ordered-slice relationships, but expert-curated labels cover only a fraction of available scans.Representation quality also needs assessment across institutions, anatomy, acquisition, and annotation practice.Source: §I · p. 1
  • L2GSSL separately regularizes local slice and global volume representations; stop-gradient lets local embeddings enter volume construction without allowing the global loss to update the local encoder.Its three-phase optimization and slice-wise execution are designed for constrained GPU memory.Source: §I · p. 1
  • L2GSSL’s largest gains occur with 10% and 25% of downstream labels and diminish with full supervision, matching its intended role under limited expert annotation.The evaluation includes LIDC-IDRI malignancy classification, direct LNDb transfer, two backbones, lesion-slice masking, and an abdominal-trauma system.Source: §I · p. 1

A. Volumetric Medical Image Learning

  • Direct 3D models expose local appearance and volumetric context simultaneously, while activation memory grows with all three spatial dimensions; slice-sequence models compress each plane before ordered-token learning, preserving throughplane context without applying every operation to a dense 3D tensor.Source: A. Volumetric Medical Image Learning · p. 2
  • Unlike encode-then-aggregate pipelines that may optimize only the final volume embedding, L2GSSL assigns separate objectives to slice and inter-slice spaces and blocks global-loss updates to the local encoder.The framework is instantiated with MST and MaxViT as a pretraining strategy, not an alternative downstream backbone.Source: A. Volumetric Medical Image Learning · p. 2

B. Self-Supervised Learning for Medical Images

Medical self-supervision includes restoration, contrastive, and volumetric spatial objectives, with choices shaped by view pairing, negative samples, and the spatial unit being embedded. L2GSSL applies separate objectives to local and global representations, with distinct gradient paths.

B. Self-Supervised Learning for Medical Images

  • Contrastive and joint-embedding methods align augmented views, while VICReg prevents collapse through batch statistics without a momentum encoder, queue, or predictor.For CT, neighboring slices can contain nearly identical anatomy, making them problematic negatives in contrastive learning.Source: B. Self-Supervised Learning for… · p. 2
  • L2GSSL applies independent self-supervised objectives to slice and volume embeddings while preventing the volume loss from updating the slice encoder.Its local/global split concerns objectives and gradient paths, not merely a hierarchical backbone.Source: B. Self-Supervised Learning for… · p. 2, Table I · p. 3

C. External Evaluation and Model-Behavior Analysis

  • External evaluation provides evidence distinct from another internal split, but does not establish prospective clinical validity.Perturbation tests show input dependence without establishing causal reasoning or clinical utility.Source: C. External Evaluation and… · pp. 2–3

A. Hierarchical Representation Spaces

  • A local encoder maps each CT slice to a token, and a global encoder aggregates the ordered tokens into a volume representation.The total objective adds a local VICReg loss for intra-slice representations and a global VICReg loss for inter-slice volumetric representations.Source: A. Hierarchical Representation Spaces · p. 3
  • Stop-gradient lets local embeddings contribute to volume construction while preventing the global objective from updating the local encoder.The spaces remain functionally coupled in the forward pass but are optimized through separate gradient paths.Source: A. Hierarchical Representation Spaces · p. 3

B. Backbone Instantiations

L2GSSL is instantiated with an explicit slice-sequence architecture and a staged convolution-attention backbone. Both designs distinguish local from global processing, while Fig. 1 depicts the framework’s directional gradient flow.

B. Backbone Instantiations

  • MST uses a ViT image encoder for local slice tokens and a slice transformer for inter-slice context, with the local encoder initialized from DINOv2 weights.MaxViT uses ImageNet-pretrained weights, assigning its stem and first two stages to the local encoder and its remaining stages to the global encoder.Source: B. Backbone Instantiations · pp. 3–4
  • Fig. 1 shows local embeddings entering the global encoder while the global loss is detached at the local encoder.Solid arrows indicate forward computation; colored arrows indicate parameter updates.Source: Figure 1 · p. 4
  • The MST and MaxViT designs test L2GSSL across an explicit slice transformer and a staged convolution-attention network.The MaxViT partition follows the backbone hierarchy rather than an explicit slice-aggregation boundary.Source: B. Backbone Instantiations · pp. 3–4

C. Slice-Wise Execution and Phased Optimization

L2GSSL processes slices in mini-batches, accumulates their embeddings in anatomical order, and trains local and global representations in three phases. This execution and schedule reduce peak activation storage and control when each representation level is optimized.

C. Slice-Wise Execution and Phased Optimization

  • Training uses local-only optimization for the first 30% of epochs, joint local and global optimization for the next 40%, and global-only optimization for the final 30%.In the final phase, the local encoder is frozen and the global encoder is optimized on its fixed embeddings.Source: C. Slice-Wise Execution and Phased… · p. 4
  • The schedule first establishes slice features, then learns inter-slice relationships while keeping updates on separate parameter paths, and finally concentrates optimization on global modeling.During the joint phase, the local encoder follows its own objective while the global encoder models ordered embeddings.Source: C. Slice-Wise Execution and Phased… · p. 4
  • Sequential slice-group processing avoids storing all slice activations simultaneously, while the global encoder operates on the compact embedding sequence.Gradient accumulation approximates a larger optimization batch, though normalization statistics are computed within each slice group when present.Source: C. Slice-Wise Execution and Phased… · p. 4

D. Peak-Activation Analysis

L2GSSL lowers peak activation storage by grouping slice-wise local computation and separating local and global activation peaks. This saves memory in the local-activation term, not necessarily execution time or throughput.

D. Peak-Activation Analysis

  • Grouped local backward passes and global encoding create activation peaks at different times, while detached embeddings persist across groups.The analysis omits parameters, optimizer state, inputs, and framework allocation to isolate activation terms affected by grouping and gradient detachment.Source: D. Peak-Activation Analysis · pp. 4–5
  • L2GSSL reduces the local-activation term, but ordered embeddings and global-encoder activations still scale with the number of slices.Slice grouping does not reduce the local-encoder forward operations per volume; sequential groups may add execution overhead, so lower peak activation storage does not imply faster training or higher data-ingestion throughput.Source: D. Peak-Activation Analysis · p. 5
  • Smaller groups reduce local-activation storage, whereas larger groups may improve within-group hardware utilization when memory permits.The group size b sets this practical trade-off.Source: D. Peak-Activation Analysis · p. 5

A. Datasets and Tasks

The evaluation uses LIDC-IDRI for controlled low-label nodule classification, LNDb for direct source transfer, and RSNA with AbdomenCT-1K for abdominal task breadth. Shared partitions and protocols distinguish label-efficiency comparisons from transfer questions.

A. Datasets and Tasks

  • LIDC-IDRI provides 1,625 nodule samples from 1,018 thoracic CT cases, with malignancy classification evaluated using 10%, 25%, or 100% of training labels.The split is made at patient/scan level before crop or slice extraction to keep related samples in one partition; LNDb is held out from pretraining, fine-tuning, and checkpoint selection.Source: A. Datasets and Tasks · p. 5

B. Experimental Setup and Evaluation Protocols

  • Protocols distinguish representation and label-efficiency evaluation on LIDC-IDRI from direct LNDb transfer and abdominal anatomy/task transfer using RSNA and AbdomenCT-1K.For downstream MST evaluation, linear probing trains only the classifier, while full fine-tuning updates the network; comparisons share the applicable backbone and pipeline and use validation AUROC for checkpoint selection.Source: B. Experimental Setup and Evaluation… · p. 5, A. Datasets and Tasks · p. 5

C. Label-Efficient Transfer on LIDC-IDRI

  • 0.0229 and 0.0294 AUROC gains over volume-level VICReg occur with 10% and 25% of labels, respectively; L2GSSL leads both AUROC and AUPRC among evaluated methods at those label fractions.With all labels, supervised MST-DINOv2 has the highest AUROC by 0.0044, while SimSiam has the highest AUPRC by 0.0080.Source: C. Label-Efficient Transfer on LIDC-IDRI · pp. 5–6
  • Linear probing shows L2GSSL improves over volume-level VICReg in both AUROC and AUPRC under identical batch size.Linear probing isolates representation quality.Source: C. Label-Efficient Transfer on LIDC-IDRI · p. 6

D. Cross-Dataset and Task Transfer

L2GSSL is evaluated for direct transfer from LIDC-IDRI to LNDb without target adaptation and for transfer to abdominal trauma prediction. It improves on the strongest evaluated self-supervised baseline in direct LNDb evaluation and yields a lower competition loss in the abdominal pipeline.

1) Direct Cross-Dataset Evaluation:

  • 0.1130 AUROC and 0.0954 AUPRC gains over volume-level VICReg occur in direct LNDb evaluation without target-domain adaptation.L2GSSL is best in both discrimination metrics in this evaluation.Source: 1) Direct Cross-Dataset Evaluation · p. 6

2) Abdominal Trauma Transfer:

  • 0.3551 severity-weighted RSNA competition loss is achieved by the best L2GSSL-expert configuration, compared with 0.3597 for the original pipeline.The configuration incorporates unlabeled, anatomically aligned AbdomenCT-1K volumes without changing downstream inference design.Source: 2) Abdominal Trauma Transfer · p. 6

1) Common GPU-Memory Budget:

L2GSSL is evaluated under a shared GPU-memory cap, against a fully 3D reference and a second backbone, and through lesion-masking and radiologist-context analyses. The results cover memory-bounded representation quality, model comparisons, lesion dependence, and sensitivity to uncertain reader judgments.

1) Common GPU-Memory Budget:

  • Under a common 47–48 GB peak-VRAM cap, L2GSSL achieves the highest AUROC and AUPRC, followed by volume-level VICReg.This comparison evaluates representation quality within the same memory envelope rather than isolating the SSL objective alone.Source: 1) Common GPU-Memory Budget · p. 6

2) Fully 3D Reference:

  • L2GSSL uses slightly less measured peak VRAM and outperforms both VoCo variants in linear probing and fine-tuning across all evaluated label fractions.The comparison is a fully 3D system-level reference, not a controlled comparison of backbone, input formulation, or initialization.Source: 2) Fully 3D Reference · p. 6

F. Backbone Portability

  • With MaxViT, L2GSSL achieves the highest AUROC and AUPRC, supporting portability beyond MST without implying universality across architectures.Source: F. Backbone Portability · p. 6

1) Lesion Dependence:

  • Masking malignant lesion-bearing slices reduces malignant true-positive rate by 73.27 percentage points and produces large AUROC and AUPRC drops.The result indicates substantial reliance on information in malignant lesion slices rather than only background or dataset-level shortcuts.Source: 1) Lesion Dependence · p. 6

2) Radiologist Context and Sensitivity to Uncertain Cases:

  • L2GSSL ROC curves shift only modestly when uncertain held-out-reader scores are retained and remain close across runs.The pooled reader operating point summarizes per-nodule judgments; a score of 3 counts as disagreement against a definite reference in the uncertainty-penalized condition.Source: 2) Radiologist Context and Sensitivity… · p. 7

H. Design and Implementation Sensitivity

The sensitivity analyses examine gradient isolation, phased optimization, and projector normalization. They find that the effects of scheduling depend on initialization, while batch normalization does not eliminate the weaker MoCo v2 and SimSiam results.

1) Component Ablation:

  • Removing stop-gradient gives the weakest discrimination under both initialization strategies; phased optimization performs best from random initialization, but removing it performs best from DINOv2 initialization.The schedule acts as an initialization-dependent optimization mechanism rather than a universally beneficial component.Source: 1) Component Ablation · p. 7, Table XI · p. 8

2) Projector Normalization Sensitivity:

  • With projector batch normalization, L2GSSL records 0.7623±0.0071 AUROC and 0.5615±0.0045 AUPRC, above MoCo v2 and SimSiam on both metrics.In slice-wise accumulation, normalization statistics are estimated within each micro-batch although gradients approximate a larger effective batch; this mismatch can produce noisy statistics and degrade training.Source: 2) Projector Normalization Sensitivity · pp. 7–8

V. DISCUSSION

The discussion reports L2GSSL's clearest gains with limited LIDC-IDRI labels and a narrower advantage under full-label fine-tuning. It relates this pattern to the local and global representation objectives and discusses memory feasibility and initialization-dependent optimization.

A. Principal Findings

  • Separating slice- and volume-level self-supervised objectives improves label-efficient learning, with the clearest gains at 10% or 25% of LIDC-IDRI labels and a narrower advantage under full-label fine-tuning.Direct LNDb evaluation tests a change in data source and annotation practice, while abdominal-trauma evaluation extends the setting across anatomy and prediction target.Source: A. Principal Findings · pp. 8–9
  • The local objective supplies view-stable slice features and the global objective organizes them across the ordered volume, an interpretation consistent with the label-fraction trend.The discussion suggests full-label fine-tuning can narrow differences among initializations because it supplies more task-specific supervision to trainable components.Source: A. Principal Findings · p. 9
  • Resource experiments show that slice-to-volume learning provides useful representations within the measured GPU-memory budgets.The discussion relates this property to high-dimensional healthcare volumes that vary in slice count while labels remain limited.Source: A. Principal Findings · p. 9
  • Removing gradient detachment gives the weakest DINOv2-initialized variant, while the full phased configuration performs best from random initialization.The ablation supports controlled local-to-global optimization while showing that the preferred schedule depends on initialization.Source: A. Principal Findings · p. 9

B. Implications for Healthcare Data Processing

L2GSSL is intended as representation pretraining, with modular processing suited to variable CT archives. Its evaluation remains bounded by retrospective datasets and unresolved clinical deployment questions.

B. Implications for Healthcare Data Processing

  • Slice-wise processing and modular boundaries support handling volumes with variable slice counts and auditing local features, sequence aggregation, and predictions separately.Source: B. Implications for Healthcare Data… · p. 9
  • Site-specific pretraining could absorb scanner or population shortcuts that de-identification and SSL alone cannot detect, so provenance and institution-level evaluation remain important.Source: B. Implications for Healthcare Data… · p. 9

C. Clinical Interpretation of the Evaluation

  • Low-label fractions do not equal clinical annotation budgets, and AUROC/AUPRC do not specify acceptable errors; decision-curve analysis and task-specific utility were not performed.Source: C. Clinical Interpretation of the… · p. 9
  • The radiologist analysis is retrospective, and LNDb transfer tests technical generalization rather than clinical transportability or whether assistance changes decisions.Source: C. Clinical Interpretation of the… · p. 9

D. Limitations

  • The VoCo comparison evaluates complete systems, not isolated objectives, because backbone, input formulation, and initialization differ.Source: D. Limitations · p. 9
  • Evaluation is retrospective CT classification and omits prospective deployment and demographic subgroup performance; resource analysis measures peak VRAM, not throughput, storage, or training time.Source: D. Limitations · pp. 9–10
  • Lesion masking supports lesion-bearing slices contributing to predictions, but input-distribution changes prevent conclusions about causal localization or reliance on other features.Source: D. Limitations · p. 10

VI. CONCLUSION

  • L2GSSL's clearest gains occur with 10% and 25% of LIDC-IDRI labels and diminish under full supervision, alongside evaluation across sources, backbones, anatomy, and tasks.Source: §VI · p. 10
Loading 2610.21356v1…