PaperlayerView exact revision on arXiv

Source-linked AI summary

Recurrent U-Net for Resource-Constrained Segmentation

Wei Wang, Kaicheng Yu, Joachim Hugonot, Pascal Fua, Mathieu Salzmann

arXiv:1906.04913v1cs.CV

TL;DR

Deep segmentation networks can be slow and require substantial training data, limiting their fit for resource-constrained settings. This paper introduces a compact recurrent U-Net and reports state-of-the-art performance on hand and retina segmentation, alongside a larger hand dataset.

  • Problem

    Deep segmentation networks can be slow and difficult to train without very large datasets, limiting their suitability for resource-constrained segmentation.

    Source: §1 · p. 1

  • Method

    The paper introduces a recurrent U-Net that iteratively refines segmentation masks and internal state while retaining U-Net compactness through parameter sharing.

    Source: §1 · pp. 1–2

  • Results

    The recurrent U-Net outperforms the state of the art on five hand-segmentation datasets and a retina vessel-segmentation dataset; Ours-DRU(4) achieves vessel-class F1 0.92 versus [17]'s 0.822.

    Source: §1 · p. 1, §4.3 · p. 8

  • Takeaways & Limitations

    The approach reaches 55 fps on 230×306 images on a TITAN X and supports trading accuracy for speed by reducing iterations.

    Source: §1 · p. 2

Abstract

from arXiv · show

State-of-the-art segmentation methods rely on very deep networks that are not always easy to train without very large training datasets and tend to be relatively slow to run on standard GPUs. In this paper, we introduce a novel recurrent U-Net architecture that preserves the compactness of the original U-Net, while substantially increasing its performance to the point where it outperforms the state of the art on several benchmarks. We will demonstrate its effectiveness for several tasks, including hand segmentation, retina vessel segmentation, and road segmentation. We also introduce a large-scale dataset for hand segmentation.

1. Introduction

The paper introduces a compact recurrent U-Net for segmentation in settings with limited training data and computational power. It reports state-of-the-art results on hand and retina segmentation, and introduces a larger hand-segmentation dataset.

1. Introduction

  • On the Keyboard Hand Dataset, ICNet is slightly faster but less accurate, while RefineNet and DeepLab are both slower and less accurate.Figure 1 plots frames per second against mIoU on a Titan X (Pascal); circle radius encodes parameter count, and the recurrent approach is shown after one, two, and three iterations.Source: Figure 1 · p. 1
  • The recurrent U-Net outperforms the state of the art on five hand-segmentation datasets and a retina vessel-segmentation dataset.The model has 0.3 million parameters, compared with 40 million for DeepLabv3+ and 118 million for RefineNet.Source: §1 · p. 1
  • The architecture iteratively refines both the segmentation mask and internal state by placing recurrent units over inner U-Net layers.Unlike a CNN+RNN with a single-layer internal state, the approach can choose which portion of its internal state to use for recursion.Source: §1 · p. 2
  • The paper evaluates the approach on hand, retina-vessel, and road segmentation, including comparisons with earlier recursive approaches and task-specific state-of-the-art methods.The authors also introduce a keyboard-typing hand dataset with 12.5K annotated images.Source: §1 · p. 2
  • The approach retains U-Net compactness and memory use through parameter sharing, reaches 55 fps on 230×306 images, and trades accuracy for speed by reducing iterations.The authors state that replacing the U-Net encoder with VGG16 yields Cityscapes performance comparable to the state of the art.Source: §1 · p. 2

2. Related Work

Compact encoder-decoder networks such as U-Net suit resource-constrained segmentation, while multi-branch architectures require balancing speed and performance. Prior recurrent segmentation methods either feed the previous mask back as input or add a recurrent unit at the network output; this work instead uses gated units spanning several U-Net layers.

2. Related Work

  • U-Net is a widely used encoder-decoder architecture, particularly in biomedical image analysis, while multi-branch models fuse features across resolutions but require careful speed-performance design.U-Net uses simpler skip connections and does not require a specific design for balancing those trade-offs.Source: §2 · p. 2

Recurrent Networks for Segmentation.

  • Earlier recurrent segmentation feeds the previous mask back with the image, or adds a standard recurrent unit at the segmentation network’s output.The paper departs from both by using gated recurrent units across several U-Net layers and feeding in previous segmentation results.Source: Recurrent Networks for Segmentation · p. 3
  • Pose-estimation refinement methods use consecutive modules without shared parameters and do not use RNN-inspired memory units to track hidden state.Those approaches therefore use more parameters than the compact-network aim described here.Source: Recurrent Networks for Segmentation · p. 3

3. Method

The method retains a compact U-Net while adding recurrent refinement of both segmentation masks and internal network states. It uses gated recurrent units spanning multiple U-Net layers and supervises predictions at each iteration.

3.1. Recurrent U-Net

  • The recurrent unit combines the previous segmentation mask with the input image and updates internal hidden states across selected U-Net encoding and decoding layers.The level ℓ determines where recurrence begins; ℓ=0 places the entire U-Net inside the recurrent unit.Source: §3.1 · p. 3

3.2. Dual-gated Recurrent Unit

  • The DRU uses update and reset gates with multilayer encoder-decoder operations, while its output block maps the new hidden state to decoding activations.The hidden state is used indirectly through recurrence, avoiding increased channel counts in the encoding and decoding layers.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3. Single-Gated Recurrent Unit

  • The SRU removes one encoder-decoder network from the DRU and incurs very little loss in segmentation accuracy.It omits the reset tensor while retaining a similar structure.Source: §3.3 · p. 4

3.4. Training

  • Training applies cross-entropy supervision at each of three recurrence iterations, with iteration weights that can emphasize the final prediction.The authors use α=1 for equal weighting or α=0.4 to put more emphasis on the final prediction.Source: §3.4 · pp. 4–5

4. Experiments

The experiments evaluate recurrent U-Net variants across hand, retina-vessel, road, and urban-landscape segmentation. Results show strong benchmark performance, while the new KBH dataset adds 12,536 annotated hand frames.

4.1. Datasets.

  • The new KBH hand dataset contains 12,536 annotated frames from 161 sequences, split by person into 30 training, 30 validation, and 101 test videos.The collection varied keyboard types, lighting, desk colors, and objects; each participant appears in only one split.Source: §4.1 · p. 5

4.2. Experimental Setup

  • The setup compares recurrent variants with task-specific state-of-the-art models, general-purpose DeepLab V3+, real-time ICNet, and U-Net-based baselines.The U-Net and recurrent baselines share parameter settings; U-Net-B uses batch normalization and U-Net-G uses group normalization.Source: §4.2 · p. 6
  • The recurrent approach also extends to the general setting by replacing the U-Net encoder with pretrained VGG-16, producing DRU-VGG16.The experiments include Cityscapes as a high-resolution urban-landscape benchmark.Source: §4.1 · p. 5, §4.2 · p. 6

4.3. Comparison to the State of the Art

  • Ours-DRU(4) performs best on average across the hand datasets, with Ours-SRU(0) a close second; DRU-VGG16 yields the overall best results with a pretrained backbone.Ours-DRU(4) beats RefineNet on four of five datasets; HOF is the exception, where the authors attribute RefineNet’s advantage to pretraining on a small dataset.Source: Table 3 · p. 7, §4.3 · p. 7
  • Ours-DRU(4) scores 0.92 on vessel-class F1 versus 0.822 reported by [17]; on roads, it exceeds U-Net-G by 8 percentage points in mIoU.DRU also yields the best retina mIoU, mPrec, and mF1, and road results exceed DeepLab V3+ and U-Net-VGG16.Source: §4.3 · p. 8, Table 4 · p. 8, Table 5 · p. 8
  • On Cityscapes, Ours-DRU is consistently better than U-Net-G and Rec-Last, while DRU-VGG16 reaches accuracy comparable to DeepLab V3.The comparison uses the validation set; the table caption notes that the method does not use multi-scaling or horizontal flips at inference.Source: §4.3 · p. 8, Table 6 · p. 8

5. Conclusion

The recurrent U-Net preserves the original U-Net’s compactness while substantially increasing performance, with recurrent units spanning several encoding and decoding layers. The paper also introduces a hand-segmentation dataset larger than existing ones and identifies broader recurrent architectures as future work.

  • Recurrent units encompass several encoding and decoding layers within the segmentation network.Source: §5 · p. 8
  • The recurrent U-Net preserves the original U-Net’s compactness while substantially increasing performance.Source: §5 · p. 8
  • The paper introduces a hand-segmentation dataset larger than existing ones.Source: §5 · p. 8
  • Future work will extend recurrent units to backbones beyond U-Net and to multi-scale recurrent architectures.Source: §5 · p. 8
Loading 1906.04913v1…