PaperlayerView exact revision on arXiv

Source-linked AI summary

DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasets

Jianpeng Zhang, Yutong Xie, Yong Xia, Chunhua Shen

arXiv:2011.10217v1cs.CV

TL;DR

Expensive voxel-level annotation leaves most datasets partially labeled, limiting joint segmentation of multiple organs and tumors. DoDNet uses task-aware dynamic filters with a single dynamic head, achieves the best overall performance across seven tasks, and transfers MOTS pretraining effectively to downstream segmentation.

  • Problem

    Expensive voxel-level annotation means most benchmark datasets contain labels for only one organ or tumor type, creating a partially labeling issue for multi-organ and tumor segmentation.

  • Method

    DoDNet combines a shared encoder-decoder, task encoding, controller-generated dynamic filters, and a single segmentation head conditioned on the input image and assigned task.

  • Results

    DoDNet achieves the best overall performance and higher inference speed on seven organ and tumor segmentation tasks, while MOTS pretraining improves downstream BCV performance from 85.30% to 86.44% average Dice.

  • Takeaways & Limitations

    MOTS-pretrained DoDNet can be transferred to downstream volumetric segmentation tasks with limited annotations.

Abstract

from arXiv · show

Due to the intensive cost of labor and expertise in annotating 3D medical images at a voxel level, most benchmark datasets are equipped with the annotations of only one type of organs and/or tumors, resulting in the so-called partially labeling issue. To address this, we propose a dynamic on-demand network (DoDNet) that learns to segment multiple organs and tumors on partially labeled datasets. DoDNet consists of a shared encoder-decoder architecture, a task encoding module, a controller for generating dynamic convolution filters, and a single but dynamic segmentation head. The information of the current segmentation task is encoded as a task-aware prior to tell the model what the task is expected to solve. Different from existing approaches which fix kernels after training, the kernels in dynamic head are generated adaptively by the controller, conditioned on both input image and assigned task. Thus, DoDNet is able to segment multiple organs and tumors, as done by multiple networks or a multi-head network, in a much efficient and flexible manner. We have created a large-scale partially labeled dataset, termed MOTS, and demonstrated the superior performance of our DoDNet over other competitors on seven organ and tumor segmentation tasks. We also transferred the weights pre-trained on MOTS to a downstream multi-organ segmentation task and achieved state-of-the-art performance. This study provides a general 3D medical image segmentation model that has been pre-trained on a large-scale partially labelled dataset and can be extended (after fine-tuning) to downstream volumetric medical data segmentation tasks. The dataset and code areavailableat: https://git.io/DoDNet

1. Introduction

Voxel-level annotation costs leave most abdominal CT datasets partially labeled, creating a challenge for learning multi-organ and tumor representations. DoDNet addresses this with a single dynamic-head network trained across such datasets and transferable to annotation-limited downstream tasks.

  • High labor and expertise costs make large-scale voxel-level annotation of multiple abdominal organs and tumors difficult.
  • Partially labeled datasets mark task-irrelevant organs and tumors as background, complicating representation learning across multiple segmentation tasks.
  • Training separate networks for each partially labeled subset increases computational complexity, while multi-head networks use shared encoders and task-specific decoders.
  • DoDNet uses a single dynamic segmentation head whose kernels are generated by a controller from the input image and assigned task.
  • DoDNet evaluates seven organ and tumor segmentation benchmarks and transfers MOTS-pretrained weights to a downstream multi-organ task.
  • The dynamic head adapts to the input and assigned task, improving the model’s stated efficiency and flexibility compared with a fixed segmentation head.

2. Related Work

Prior work addresses partially labeled segmentation with separate networks, multi-head designs, or dynamic filtering, but DoDNet combines task-aware dynamic filtering with a single flexible head for organs and tumors.

  • Partially labeled medical image segmentation: Training multiple networks for separate partially labeled datasets wastes computational resources and scales poorly.
  • Partially labeled medical image segmentation: Some methods co-train heterogeneous networks using task-shared encoders and task-specific decoders across multiple segmentation tasks.
  • Partially labeled medical image segmentation: DoDNet treats each organ as a single-class task rather than labeling organs absent from a dataset as background, because they may be foreground elsewhere.
  • Partially labeled medical image segmentation: Unlike multi-head approaches, DoDNet uses one flexible dynamic head and applies task priors there rather than embedding them in the encoder and decoder.
  • Dynamic filter learning: Dynamic filter learning has been used for flexibility, representation capacity, cross-modal fusion, and instance segmentation, whereas DoDNet targets partially labeled 3D medical segmentation.
  • Dynamic filter learning: DoDNet’s dynamic filters depend on both the input image and assigned task, extending prior input-conditioned dynamic-filter designs.

3. Our Approach

DoDNet formulates partially labeled multi-organ and tumor segmentation as a single-network problem. Its shared encoder-decoder, task encoding, controller, and dynamic head generate task- and image-conditioned predictions while ignoring unavailable labels during training.

  • 3.1. Problem definition: The problem involves m partially labeled datasets, each providing annotations for a specific organ-and-tumor segmentation task.The task set includes examples such as liver and tumor segmentation and kidney and tumor segmentation.
  • 3.1. Problem definition: DoDNet replaces separate task networks with one network f trained across partially labeled datasets.The proposed formulation explicitly targets a single network rather than independently parameterized networks for each dataset.
  • 3.2. Encoder-decoder architecture: DoDNet combines a shared encoder-decoder, task encoding module, dynamic filter generation module, and dynamic segmentation head.The shared encoder-decoder produces task-independent semantic features containing information about multiple organs and tumors.
  • 3.2. Encoder-decoder architecture: The shared encoder-decoder uses an U-like architecture with four encoder downsampling operations and four decoder upsampling operations.The encoder uses 3D residual blocks, while the decoder combines upsampled features with corresponding low-level encoder features.
  • 3.3. Task encoding: A task prior is encoded as an m-dimensional one-hot vector and concatenated with aggregated features for dynamic filter generation.Unlike extending the task vector spatially as additional input channels, DoDNet uses it after feature aggregation to reduce computational and spatial complexity.
  • 3.4. Dynamic filter generation: The controller generates convolution kernels from globally averaged image features and the assigned task, enabling task-specific dynamic heads.The dynamic head contains three stacked 1 × 1 × 1 convolutional layers and generates 162 parameters for organ and tumor predictions.
  • 3.6. Training and Testing: Training jointly uses Dice and binary cross-entropy losses while ignoring predictions for targets whose annotations are unavailable.For colon tumor segmentation, the organ prediction is excluded from loss computation and error back-propagation because organ annotations are absent.
  • 3.6. Training and Testing: At inference, DoDNet generates kernels for an assigned task and can produce separate kernel groups when all m tasks are required.The generated dynamic head segments the organ and tumors specified by the task.

4. Experiment

Experiments evaluate DoDNet on the MOTS partially labeled tasks, ablate dynamic-head design choices, compare competing methods, and test MOTS pre-training for downstream segmentation.

  • Experiment setup: MOTS contains seven partially labeled organ and tumor segmentation tasks, with 920 training and 235 test abdominal CT scans.The dataset includes 1,155 scans collected from various clinical sites and resliced to 1.5 × 0.8 × 0.8mm3 voxels.
  • Ablation study: The ablation study evaluates dynamic-head depth, width, and the image-feature and task-encoding conditions used for dynamic-filter generation.Validation averages Dice and HD over 11 organs and tumors.
  • Ablation study: Depth 3 and width 8 provide the selected dynamic-head configuration, while performance is robust across depths and stabilizes when width is at least 8.Increasing width from 4 to 8 improves performance substantially, whereas increasing it from 8 to 16 slightly reduces performance.
  • Comparing to state-of-the-art methods: 75.64% averaged Dice and 19.50 averaged HD are achieved by DoDNet across 11 categories on seven partially labeled datasets.The comparison includes individual networks, multi-head networks, and single-network conditional methods.
  • Comparing to state-of-the-art methods: DoDNet outperforms other methods qualitatively, especially for small tumors, while single-network approaches use about 17.3M parameters and repeated inference affects several competitors’ speed.Multi-Nets requires seven networks, whereas DoDNet shares the encoder-decoder and specializes its dynamic head by task.
  • MOTS Pre-training for downstream tasks: MOTS pre-training produces the fastest convergence and a remarkable validation-performance boost compared with training from scratch or pre-training on the smaller Hepatic Vessel dataset.Validation performance is measured by averaged Dice over 13 categories.
  • MOTS Pre-training for downstream tasks: 86.44% average Dice, 1.17 average mean surface distance, and 15.62 average HD are obtained on the BCV unseen test set with MOTS pre-training.Compared with training from scratch, MOTS pre-training improves Dice from 85.30% to 86.44% and reduces average mean surface distance from 1.46 to 1.17 and average HD from 19.67 to 15.62.

5. Conclusion

The paper proposes DoDNet and MOTS for multi-organ and tumor segmentation from partially labeled abdominal CT data. Experiments report strong performance across seven tasks, higher inference speed, and useful transfer to annotation-limited downstream tasks.

  • 5. Conclusion: DoDNet uses a single encoder-decoder network with a dynamic head to address partially labeled multi-organ and tumor segmentation.The paper also introduces the large-scale partially labeled MOTS dataset.
  • 5. Conclusion: Task encoding and dynamic filter learning support the best overall performance across seven organ and tumor segmentation tasks and higher inference speed than competitors.The conclusion attributes these outcomes to the proposed task encoding and dynamic filter learning.
  • 5. Conclusion: MOTS-pre-trained weights transfer successfully to downstream tasks with limited annotations, providing a pre-trained 3D network for small-sample 3D medical image segmentation.The conclusion presents this transfer as evidence of the value of DoDNet and MOTS.
Loading 2011.10217v1…