PaperlayerView exact revision on arXiv

Source-linked AI summary

Mumford-Shah Loss Functional for Image Segmentation with Deep Learning

Boah Kim, Jong Chul Ye

arXiv:1904.02872v2cs.CVcs.LGstat.ML

TL;DR

Supervised CNN segmentation requires costly pixel-level labels, while classical Mumford–Shah methods avoid labels but are computationally expensive and limited for semantic segmentation. The paper introduces a neural-network loss based on Mumford–Shah, using softmax outputs as differentiable characteristic functions. Experiments show it supports semi-supervised and unsupervised segmentation and can regularize supervised semantic segmentation.

  • Problem

    Deep segmentation methods usually require large amounts of high-quality pixel-wise labels, whereas classical label-free Mumford–Shah methods are computationally expensive and limited in semantic segmentation.

  • Method

    The paper directly minimizes a Mumford–Shah-based loss in neural networks by treating softmax outputs as differentiable approximations of characteristic functions.

  • Results

    The proposed loss enables semi-supervised and unsupervised segmentation and can enhance supervised semantic segmentation as a regularized function.

  • Takeaways & Limitations

    Mumford–Shah pixel-statistics information can be integrated into CNN training to use unlabeled or sparsely labeled data for image segmentation.

Abstract

from arXiv · show

Recent state-of-the-art image segmentation algorithms are mostly based on deep neural networks, thanks to their high performance and fast computation time. However, these methods are usually trained in a supervised manner, which requires large number of high quality ground-truth segmentation masks. On the other hand, classical image segmentation approaches such as level-set methods are formulated in a self-supervised manner by minimizing energy functions such as Mumford-Shah functional, so they are still useful to help generation of segmentation masks without labels. Unfortunately, these algorithms are usually computationally expensive and often have limitation in semantic segmentation. In this paper, we propose a novel loss function based on Mumford-Shah functional that can be used in deep-learning based image segmentation without or with small labeled data. This loss function is based on the observation that the softmax layer of deep neural networks has striking similarity to the characteristic function in the Mumford-Shah functional. We show that the new loss function enables semi-supervised and unsupervised segmentation. In addition, our loss function can be also used as a regularized function to enhance supervised semantic segmentation algorithms. Experimental results on multiple datasets demonstrate the effectiveness of the proposed method.

I. INTRODUCTION

Deep segmentation achieves strong performance but typically needs extensive pixel-level annotation, while classical Mumford–Shah methods avoid labels at higher computational cost and with semantic-segmentation limitations. The paper proposes directly minimizing Mumford–Shah with neural networks by using softmax outputs as differentiable characteristic functions.

  • Motivation: Deep CNN segmentation requires large datasets with high-quality pixel-wise labels, whereas classical variational methods can produce pixel-wise predictions without ground truth.Classical methods minimize energy functions such as Mumford–Shah, but their computational cost and semantic-segmentation limitations motivate a hybrid approach.
  • Proposed approach: The proposed loss incorporates Mumford–Shah pixel-similarity information into CNN segmentation, supporting supervised, semi-supervised, and unsupervised training.It acts as a regularizer with semantic labels and as a self-supervised loss without them.
  • Proposed approach: Softmax outputs provide a differentiable approximation to Mumford–Shah characteristic functions, enabling direct neural-network minimization instead of level-set evolution.Deep networks perform the minimization in a data-driven way and support real-time computation.
  • Mumford–Shah formulation: The Mumford–Shah energy models an image as N piecewise-constant regions using class characteristic functions and representative average values.The formulation extends to multichannel images by treating image measurements and class values as channel-wise vectors.
  • Classical methods: Multiphase level-set methods make the non-differentiable characteristic-function energy differentiable through level functions and can segment spatially separated regions of the same class.Their Euler–Lagrangian formulation uses vector Heaviside functions and approximated Dirac deltas.

1) Supervised Semantic Segmentation:

The paper positions its method as a way to reduce annotation demands in semantic segmentation while retaining CNN-based processing. It directly minimizes the Mumford–Shah functional during training and uses unlabeled images and image statistics as additional information.

  • Motivation: Pixel-level annotation is difficult to obtain at the scale required by supervised semantic segmentation.This motivates methods that use weaker supervision or unlabeled data.
  • Contribution: The proposed method directly minimizes Mumford–Shah by back-propagation rather than relying on computationally expensive level-function evolution.The loss is added to an existing CNN segmentation approach.
  • Contribution: Unlabeled images can participate in training without weak labels because the Mumford–Shah functional depends on pixel statistics.This reduces the manual annotation requirement for semi-supervised learning.
  • Contribution: The method avoids the complex pre- and post-processing often used by CNN-based unsupervised segmentation methods.Its implementation requires adding the Mumford–Shah loss to an existing CNN approach.
  • Contribution: As a data-adaptive regularizer, the loss helps fully supervised networks adapt to image-specific statistics and further improve segmentation performance.This use complements the method’s semi-supervised and unsupervised applications.

III. THEORY

The paper uses CNN softmax outputs as differentiable approximations of Mumford–Shah characteristic functions, enabling a CNN-compatible loss that can be optimized by backpropagation and used without labels.

  • Key observation: CNN softmax outputs approximate characteristic functions because each class channel approaches one when a pixel belongs to that class.This provides differentiable class-membership representations for direct Mumford–Shah minimization.
  • Loss formulation: The proposed CNN-inspired Mumford–Shah functional uses the input image, softmax outputs, class-average pixel values, and learnable network parameters.The formulation is introduced as a loss function based on the Mumford–Shah functional.
  • Optimization and supervision: The loss is differentiable with respect to network parameters and can therefore be minimized by backpropagation during training.Its differentiability also supports self-supervised segmentation when label-based loss is unavailable.
  • Optimization and supervision: The proposed loss provides a self-supervised objective for unsupervised segmentation and uses pixel-value distributions to augment semantic segmentation.This extends the loss beyond fully supervised training.

B. Learning Minimizers of Mumford-Shah Functional

The network is interpreted as learning or unrolling minimization steps for the Mumford–Shah functional, and the resulting loss can be added to supervised, semi-supervised, or unsupervised segmentation pipelines.

  • Learning minimizers: The Mumford–Shah Euler–Lagrange equation yields a fixed-point iteration with iteration index k, a discrete Dirac delta, and step size η^k.The iteration is presented as a way to obtain solutions of the Euler–Lagrange equation.
  • Learning minimizers: The neural network can be interpreted as a finite unrolling of these iterations, inspired by the Learned iterative soft-thresholding algorithm.The unrolled network is truncated to a fixed number of iterations and compared with level-set methods for unsupervised segmentation.
  • Applications: The proposed loss can be combined with existing supervised, semi-supervised, and unsupervised segmentation algorithms.The underlying segmentation network may accept images with or without pixel-level annotations.
  • Semi-supervised learning: In semi-supervised training, labeled images use segmentation and Mumford–Shah losses, while unlabeled images update network parameters by minimizing the Mumford–Shah loss.The proposed flow supports mixed labeled and unlabeled inputs without estimated or weak supervisory labels.
  • Semi-supervised learning: For unlabeled inputs, training uses only the Mumford–Shah loss, whereas labeled inputs use it together with the conventional segmentation loss.With full pixel-wise annotations, both losses are used so the network considers pixel similarity and semantic information.

2) In the Absence of Semantic Labels:

The self-supervised Mumford-Shah loss enables unsupervised segmentation, but intensity inhomogeneity can split a single region into multiple segments. Bias-field estimation adds supervision that may mitigate this limitation and can be jointly predicted with segmentation.

  • Setting α = 0 enables unsupervised segmentation based on pixel statistics.
  • Intensity inhomogeneities can cause the Mumford-Shah functional to produce separate regions within one semantic region.
  • Bias-field estimation can provide additional supervision when unsupervised semantic loss is unavailable.
  • The network can jointly estimate a bias field and segmentation maps by minimizing the modified loss.

IV. METHOD

The experiments span natural-image, medical-image, and boundary-based segmentation datasets, using supervised, semi-supervised, or unsupervised training according to label availability.

  • PASCAL VOC 2012 and SBD provide an augmented natural-image training set of 10,582 images, evaluated on 1,449 validation images.
  • LiTS contains 201 contrast-enhanced 3D CT scans, including 131 scans with pixel-wise liver and tumor labels used for training and inference.
  • BRATS contains 274 multimodal MRI scans with manual segmentation labels, and 249 training scans were used for complete tumor segmentation.
  • BSDS500 provides 200 training, 100 validation, and 200 test RGB images with human-annotated labels.
  • Datasets with pixel-level labels were trained semi-supervisedly, whereas datasets without them were trained unsupervisedly using PyTorch.

1) Semi-supervised Object Segmentation in Natural Images:

The study evaluates the proposed loss across semi-supervised, unsupervised, and comparative segmentation settings, using established architectures and controlled labeled-data ratios.

  • PASCAL VOC experiments compare SSGAN and AdvSemi with or without the proposed loss against a modified DeepLab-v2 baseline.
  • The medical segmentation experiments use modified U-Net models for LiTS and BRATS, with three adjacent slices as input and Adam optimization.
  • The proposed loss converts U-Net to unsupervised segmentation on BSDS500 and is added to Backprop to test improvement of an existing unsupervised network.
  • The cnnLevelset comparison uses PASCAL VOC images with weak bounding-box labels only for preprocessing, while segmentation is trained unsupervisedly.
  • The semi-supervised experiments randomly use 1/2, 1/4, or 1/8 of images as labeled data, with the remainder treated as unlabeled.

1) Experimental Scenario:

Experiments assess segmentation across varying labeled-data amounts and compare the proposed loss with baselines on natural and medical-image tasks. The reported results show improved object segmentation and broad class-level gains, including with limited labels.

  • The proposed loss produced more accurate object segmentation than AdvSemi across increasing labeled-data amounts and still guided segmentation with few labels.
  • Table I reports IoU for all PASCAL VOC classes using 1/4 labeled data, with the baseline defined as fully supervised training on the same labeled subset.
  • Table II compares IoU, accuracy, and recall on PASCAL VOC using 1/4 labeled data.
  • Table III compares mIoU across semi-supervised methods and labeled-data ratios on PASCAL VOC.
  • The proposed method improved most of the 20 PASCAL VOC classes, although car, chair, and motorbike did not improve in the reported comparison.
  • LiTS experiments use 1/23, 1/10, and 1/3 labeled scans, while BRATS uses 1/4 labeled scans and varies β under the same training condition.

2) Qualitative Evaluation:

The proposed Mumford–Shah loss improves semi-supervised tumor segmentation across labeled-data regimes and can also regularize fully supervised training. It enhances boundary and small-tumor delineation while achieving strong BRATS performance.

  • Qualitative tumor segmentation: IoU improved from below 0.5 to above 0.7 in several cases when the proposed loss replaced the original deep-network training.The pixel-level loss also improved boundary detail and helped detect tiny and thin tumors.
  • Semi-supervised evaluation: 10% to 20% performance improvement was observed on LiTS across different labeled-data ratios using the semi-supervised Mumford–Shah loss.Evaluation used Dice, precision, recall, and IoU on 13 LiTS validation cases.
  • Supervised regularization: The proposed loss improved segmentation even when all training data were labeled, functioning as a regularizer in supervised training.This corresponds to a labeled-data ratio of 1.
  • Semi-supervised evaluation: 88% Dice and 79% IoU were achieved on BRATS with β = 10^-7 and one-quarter labeled training data.The reported sensitivity analysis found similar improvements across various non-zero β values, with β = 10^-7 producing the best scores.

2) Qualitative Evaluation on BSDS500:

On BSDS500 and related object-segmentation comparisons, the proposed loss improves unsupervised segmentation, supports U-Net without ground-truth labels, and balances semantic coherence with boundary detail. The method also reduces computational time and benefits complicated foregrounds through bias-field estimation.

  • BSDS500 qualitative evaluation: The proposed loss improved existing Backprop segmentation and converted U-Net into an unsupervised method with state-of-the-art performance.The experiments included foreground objects such as humans and animals.
  • BSDS500 quantitative evaluation: U-Net with the proposed loss outperformed conventional level-set segmentation by 7% in Region Covering without ground-truth labels.Modified Backprop with the Mumford–Shah loss gained 6% in Region Covering over original Backprop.
  • Object-segmentation comparison: The proposed method balanced cnnLevelset’s coarse maps and CRF’s detailed but semantically distant edges, producing maps more related to object semantics.The comparison was reported for unsupervised object segmentation on PASCAL VOC 2012.
  • Efficiency and bias correction: The proposed algorithm used significantly less computational time than the comparison methods.Bias-field estimation also helped segment complicated foreground objects by removing unnecessary details.
Loading 1904.02872v2…