PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcing intra-operative tumour localisation with weak supervision from pre-operative appearances

Sidhart Sami, Ali Sharjeel, Sri Atluri, Miao Zhao, Caroline Roney, Eirini Marouli, Shaheer U. Saeed

arXiv:2610.31253v1cs.CV

TL;DR

Limited tumour visibility on intra-operative CT and challenges transferring pre-operative delineations make HCC localisation difficult. The paper uses MR-trained classifier probabilities to guide an RL search on CT, comparing direct, few-shot, and synthetic transfer. Synthetic transfer outperformed the other evaluated methods, though boundary delineation remained challenging.

  • Problem

    HCC can be difficult to distinguish on intra-operative CT, while scarce paired MR–CT data, deformation, and costly tumour labels constrain registration-based transfer and direct CT learning.

    Source: §1 · p. 1

  • Method

    An MR-trained tumour-presence classifier scores intra-operative image windows, and an RL agent uses those probabilities as weak guidance, with direct, few-shot, and synthetic-MR transfer strategies evaluated.

    Source: §1 · p. 2

  • Results

    Synthetic transfer significantly outperformed other evaluated methods, achieving mean Dice 0.31 ± 0.10 and 93% localisation success, while HCC boundary delineation remained challenging.

    Source: §5 · p. 4, §4.1 · p. 4

  • Takeaways & Limitations

    The results demonstrate the potential to transfer learned tumour appearances across modalities for CT localisation without paired MR–CT training images.

    Source: §5 · p. 4

    The modality gap may affect absolute classifier probabilities, so the method relies on tumour-containing windows retaining higher probabilities on average than non-tumour windows.

    Source: §2.2 · p. 2

Abstract

from arXiv · show

Limited tumour visibility on intra-operative images complicates surgical navigation, including liver hepatocellular carcinoma (HCC) targeting on computed tomography (CT) imaging. This makes labelled datasets difficult to curate for deep learning, and transferring pre-operative delineations through registration remains challenging due to intra-operative deformation and lack of paired pre- and intra-operative data. We propose to transfer tumour appearances learned from pre-operative magnetic resonance (MR) images to guide localisation on intra-operative CT through weak supervision. A classifier is trained to classify tumour presence on pre-operative images. This is applied to windows on intra-operative images, and despite the domain gap, it assigns higher tumour-presence probabilities on average to tumour-containing windows. These weak cross-domain signals guide a reinforcement learning agent to localise tumours on intra-operative images. We compare direct transfer, few-shot adaptation and synthetic transfer (trained using unpaired MR and CT), which translates CT into synthetic MR before localisation. On an external HCC cohort, synthetic transfer outperformed other tested methods and CT-only baselines. Our framework enables cross-modality tumour localisation without paired MR-CT training images.

1. INTRODUCTION

Pre-operative MR supports tumour detection, but limited tumour visibility on intra-operative CT makes HCC localisation and labelled-data curation difficult. The paper proposes weakly transferring MR-learned tumour appearance to guide tumour search on CT.

1. INTRODUCTION

  • Pre-operative MR provides soft-tissue contrast for tumour detection and delineation, whereas intra-operative CT is rapidly acquired for navigation but may show HCC poorly against surrounding liver.The paper frames these modalities as complementary imaging used at different stages of care.Source: §1 · p. 1
  • Registration and direct CT learning are constrained by scarce paired MR–CT data, intra-operative deformation, and the expert effort needed to label low-visibility tumours.Reported tumour Dice scores of 0.16–0.19 contrast with anatomical-structure Dice of 0.72 for registration approaches.Source: §1 · p. 1
  • The framework applies an MR-trained tumour-presence classifier to intra-operative image windows and uses its probabilities as reward signals for an RL agent searching for tumour locations.Window probabilities are spatially aggregated into a tumour score map and thresholded to obtain a segmentation.Source: §1 · p. 2
  • The study compares zero-shot classifier transfer, few-shot fine-tuning on CT examples, and synthetic transfer after translating CT into synthetic MR with a CycleGAN trained on unpaired images.The framework does not require paired images or tumour segmentation labels in the zero-shot setting.Source: §1 · p. 2

2. METHODS

The method trains an MR tumour-presence classifier, applies it to intra-operative CT windows, and uses its probabilities to guide reinforcement-learning localisation. It evaluates direct, few-shot, and synthetic transfer, with synthetic transfer localising on translated MR-like images.

2.1. Tumour-presence classifier training

  • The classifier is trained on labelled MR images to predict tumour presence, using binary cross-entropy and requiring presence labels rather than voxel-wise delineations.Source: §2.1 · p. 2

2.2. Window-based weak supervision

  • The MR-trained classifier scores local CT windows as relative weak signals; a voxel map retains the maximum score among visited windows covering each voxel before thresholding.The threshold is selected by grid search.Source: §2.2 · pp. 2–3

2.3. Reinforcement learning for tumour localisation

  • The agent navigates CT windows sequentially, receiving rewards from changes in classifier probability and learning a policy that favours higher-probability windows.The classifier remains fixed during policy optimisation; visited-window probabilities are aggregated at inference.Source: §2.3 · p. 3

2.4.1. Zero-shot transfer

  • In zero-shot transfer, the fixed MR-trained classifier is applied directly to CT windows without labelled CT examples for classifier adaptation.Source: §2.4.1 · p. 3

2.4.2. Few-shot adaptation

  • The MR-trained classifier is fine-tuned on a small set of CT inputs with binary tumour-presence labels, then fixed for reinforcement-learning localisation.Fine-tuning uses the same binary cross-entropy objective, initialized with the MR-trained parameters.Source: §2.4.2 · p. 3

2.4.3. Synthetic image transfer

  • The synthetic-transfer strategy translates CT into synthetic MR with an unpaired CycleGAN, then uses the fixed MR-trained classifier for localisation and maps scores back to CT coordinates.The training objective combines adversarial losses with a cycle-consistency term weighted by λ.Source: §2.4.3 · p. 3

3. EXPERIMENTS

The experiments combine MR and CT datasets for development and validation, then evaluate localisation on an external, radiologist-annotated HCC cohort. They compare transfer strategies with CT-only and registration methods, reporting patient-level Dice and localisation success rate.

3.1. Data

  • The MR dataset comprised 334 patients and the CT datasets comprised 313 patients, with volumes resampled and normalised for model use.MR volumes were resampled to 1.0 mm isotropic spacing and cropped to 128×128×128 voxels; CT volumes were resampled to 1.5 mm isotropic spacing.Source: §3.1 · p. 3
  • Evaluation used TCIA HCC-TACE-Seg (105 patients) with radiologist voxel-wise tumour annotations as ground truth.Development and validation sets were split 70%/30% at the patient level.Source: §3.1 · p. 4

3.2. Networks and hyperparameters

  • The classifier, CycleGAN, and RL agent used SE-ResNet3D, 3D encoder–decoder generators with multi-scale patch discriminators, and PPO, respectively.On one Nvidia Quadro GV100 GPU, training took 8, 16, and 48 hours, respectively.Source: §3.2 · p. 4

3.3. Experimental setup

  • Comparisons included zero-shot and few-shot transfer, synthetic transfer, registration methods, and CT-only baselines using full or weak supervision.Few-shot fine-tuned on 40 development CT volumes; registration results used a different evaluation cohort, while patient-level Dice and localisation success rate were reported.Source: §3.3 · p. 4

4. RESULTS

Synthetic transfer achieved the strongest reported comparison, while sensitivity analyses examined window size and segmentation threshold. Qualitative results show images alongside ground-truth and predicted segmentations.

4.1. Comparisons

  • Figure 3 presents qualitative images above ground-truth segmentations in green and smoothed predictions in red.The caption specifies the display encoding but reports no quantitative comparison.Source: Figure 3 · p. 4
  • 0.31 ± 0.10 mean Dice and 93% success rate were achieved by synthetic transfer, with significantly higher Dice than all other methods (all p < 0.01).Few-shot transfer did not differ significantly in Dice from the CT-only nn-UNet FS baseline (p > 0.11); registration results were contextual because they used a different cohort.Source: Table 1 · p. 4, §4.1 · p. 4

4.2. Sensitivity analysis

  • A 32 × 32 × 16 voxel window achieved the highest mean Dice (0.31 ± 0.10), closely followed by 32 × 32 × 32.The sensitivity analysis also found that mean Dice rose with threshold τ from 0.90 to 0.98, then decreased slightly at 0.99.Source: §4.2 · p. 4, Table 2 · p. 4

5. CONCLUSION

The study presents weakly supervised transfer of tumour appearance from MR to intra-operative CT without paired MR–CT training images. Synthetic transfer performed best among evaluated methods, although boundary delineation remained challenging.

5. CONCLUSION

  • Synthetic transfer significantly outperformed the other evaluated methods, which the authors suggest may reflect stronger MR-trained classifier guidance after reducing the modality gap.Comparatively poorer CT-only results highlight the value of MR-derived information.Source: §5 · p. 4
  • Despite a 93% localisation success rate, the 0.31 Dice score indicates that HCC boundary delineation remains challenging.The authors identify alternative supervision as a potential avenue to explore.Source: §5 · p. 4
  • The framework uses an MR-trained classifier to guide CT tumour localisation without paired MR–CT training images.The authors describe this as demonstrating the potential of transferring learned tumour appearances across modalities through weak supervision.Source: §5 · p. 4

Compliance with Ethical Standards

The study used retrospective human-subject data made available in open access and reports informed consent and ethical compliance for its internal dataset.

Compliance with Ethical Standards

Related papers

Loading 2610.31253v1…