Source-linked AI summary
HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation
Yuval Nirkin, Lior Wolf, Tal Hassner
TL;DR
Real-time semantic segmentation seeks strong accuracy without sacrificing speed. HyperSeg uses an encoder that generates spatially varying decoder weights through a nested hypernetwork and dynamic patch-wise convolutions, achieving SotA accuracy–runtime trade-offs across multiple benchmarks. The method is efficient overall but incurs a runtime penalty from unoptimized dynamic convolution operations.
Problem
Real-time semantic segmentation requires balancing accuracy and speed, while local adaptation is desirable because different image regions may contain different objects.
Method
HyperSeg combines an encoder, nested U-Net context head, weight-mapping modules, and a decoder with locally connected blocks whose weights vary by input and spatial patch.
Results
The method achieves SotA accuracy–runtime trade-offs across PASCAL VOC 2012, Cityscapes, and CamVid, including 79.1% mIoU at 16.6FPS on Cityscapes without additional training data.
Takeaways & Limitations
HyperSeg demonstrates that hypernetworks with local dynamic decoder weights can support real-time semantic segmentation across multiple benchmarks.
Takeaways & Limitations
The model incurs a large runtime penalty because its dynamic patch-wise convolution operations are unoptimized.
Abstract
from arXiv · showhide
We present a novel, real-time, semantic segmentation network in which the encoder both encodes and generates the parameters (weights) of the decoder. Furthermore, to allow maximal adaptivity, the weights at each decoder block vary spatially. For this purpose, we design a new type of hypernetwork, composed of a nested U-Net for drawing higher level context features, a multi-headed weight generating module which generates the weights of each block in the decoder immediately before they are consumed, for efficient memory utilization, and a primary network that is composed of novel dynamic patch-wise convolutions. Despite the usage of less-conventional blocks, our architecture obtains real-time performance. In terms of the runtime vs. accuracy trade-off, we surpass state of the art (SotA) results on popular semantic segmentation benchmarks: PASCAL VOC 2012 (val. set) and real-time semantic segmentation on Cityscapes, and CamVid. The code is available: https://nirkin.com/hyperseg.
1. Introduction
HyperSeg introduces a hypernetwork-based encoder-decoder for semantic segmentation that adapts decoder weights both to each input and spatial patch. It targets real-time accuracy–runtime trade-offs and reports SotA results across major benchmarks.
- Real-time semantic segmentation must balance accuracy with processing speed for applications including mobile and augmented-reality systems.
- HyperSeg uses an internal U-Net to map encoded features to dynamic decoder weights, whose values vary spatially across image patches.The design provides local adaptation because different image regions may contain different objects.
- The proposed method achieves SotA accuracy-versus-runtime trade-offs on PASCAL VOC 2012, Cityscapes, and CamVid.Cityscapes and CamVid SotA accuracy results are obtained under real-time conditions.
- The architecture combines a U-Net within a U-Net with dynamic patch-wise convolutions whose weights vary per input and spatial location.
2. Related work
Prior segmentation methods emphasize efficient encoder-decoder designs, receptive-field expansion, attention, and feature fusion. HyperSeg extends this landscape by combining hypernetworks with spatially varying locally connected layers for semantic segmentation.
- Hypernetworks: Hypernetworks generate weights for primary networks and had not previously been proposed for semantic segmentation, according to the authors.
- Locally connected layers: Locally connected layers remove weight sharing while preserving a spatial connectivity pattern similar to convolutional layers.
- Locally connected layers: Earlier locally connected applications modeled different face regions separately, while later work explored partial sharing within image patches.
- Locally connected layers: The authors claim HyperSeg is the first method combining locally connected layers with hypernetworks for semantic segmentation or image-to-image mapping.
- Semantic segmentation: Dilated convolutions and spatial pyramid pooling expand receptive fields, while attention methods model scale or long-range dependencies at differing computational costs.
- Real-time segmentation: Real-time segmentation methods typically pair an efficient encoder backbone with a relatively small decoder to trade accuracy against computation.
- Semantic segmentation: Efficient segmentation systems include encoder-decoder, image-pyramid, feature-fusion, and lightweight convolutional designs such as SegNet, ENet, ICNet, GUNet, and SwiftNet.
3. Method
HyperSeg uses a hypernetwork encoder-decoder in which context features generate decoder weights dynamically, with weights varying across spatial patches. Its nested context head and block-local weight mapping support adaptive decoding while controlling memory and computation.
- 3. Method: The architecture combines a backbone, nested-U-Net context head, and decoder primary network connected by encoder-decoder skip connections.The context head and weight mapper form the hypernetwork components, while the primary network acts as the decoder.
- 3. Method: Decoder meta-block weights are generated dynamically from encoder-derived signals and vary across spatial locations.The decoder receives feature maps, positional encodings, and generated weights to produce the segmentation prediction.
- 3.3. Dynamic patch-wise convolution: Dynamic patch-wise convolutions apply location-specific weight grids to feature-map patches, with neighboring pixels wrapped into each padded patch.The meta block uses these convolutions in place of regular convolutions and can omit depth-wise and final point-wise layers for very small patches.
- 3.1. The encoder and the hypernetwork: The context head combines information from multiple patches to address the limited receptive field of individual backbone patches.It uses a nested U-Net with downsampling, bottom-level average pooling, nearest-neighbor upsampling, concatenation, and fully connected layers.
- 3.2. The decoder (the primary network): Weight-mapping layers are embedded in decoder meta-blocks so weights are generated immediately before consumption, reducing peak memory expansion.The decoder contains n + 1 meta blocks with bilinear upsampling and concatenation, and each block is based on an inverted residual design.
4. Experimental results
The experiments evaluate HyperSeg across three benchmarks using standard accuracy, speed, computation, and parameter metrics. Results report strong accuracy–runtime trade-offs, while also identifying runtime costs from unoptimized dynamic operations and qualitative failure cases.
- The evaluation covers PASCAL VOC 2012, Cityscapes, and CamVid using mIoU, FPS, GFLOPs, and trainable parameters.
- PASCAL VOC 2012: HyperSeg achieves the best PASCAL VOC validation mIoU with lower GFLOPs and fewer trainable parameters than previous work.
- Cityscapes: On Cityscapes, the models achieve the best validation and test accuracy among fast methods, together with the best accuracy–runtime trade-off.The comparison includes methods running at 10 FPS or faster and reporting test-set mIoU.
- Cityscapes: Cityscapes GFLOPs–accuracy results show a significantly better trade-off than previous methods, although unoptimized DPWConv operations impose a large runtime penalty.GFLOPs indicate potential runtime performance after functions are optimized, but do not directly correlate with FPS.
- CamVid: On CamVid, both HyperSeg variants achieve state-of-the-art mIoU, with HyperSeg-S running at 38 FPS.HyperSeg-S uses 768×576 resolution, while HyperSeg-L uses 1024×768 resolution.
5. Conclusions
The paper combines autoencoders with hypernetworks to create a U-Net that adapts segmentation processing dynamically and locally. Experiments report state-of-the-art performance across multiple benchmarks.
- The proposed hypernetwork combines an encoder backbone, an internal U-Net context head, and multiple weight-mapping heads for a multi-block decoder.Decoder blocks use locally connected layers whose weights are generated dynamically.
- The resulting U-Net dynamically and locally adapts segmentation to the input image, and outperforms state-of-the-art methods across multiple benchmarks.
A. Feature division algorithm
The feature division algorithm partitions channels into units compatible with each weight’s group size, then allocates units proportionally across weights.
- Channels are divided into units of size su, set to the maximum group width so divided channels remain divisible by each gwi.The group widths gwi are powers of two in the experiments.
- Each weight receives one unit initially, and remaining units are allocated proportionally to weight totals starting with larger weights.Smaller weights receive the remainder from rounded-down allocations.
B. Model details
The model details specify reduction factors, channel configurations, convolution mappings, and dataset-dependent training losses across the HyperSeg variants.
- Reduction factors ri correspond to feature maps Fi from the backbone, while group widths gwi are fixed or experimentally adapted by model and dataset.The earliest HyperSeg-L PASCAL VOC model uses gwi = 16 at all levels.
- A single channel arrow denotes a 1×1 convolution, while two arrows specify the input, hidden, and output channels of a full meta block.
- The PASCAL VOC model uses cross-entropy loss, whereas the other models use bootstrapped cross-entropy loss.
C. Additional ablation studies
The ablations test spatially varying convolutions, positional encoding, and patch-grid size, showing accuracy–speed trade-offs across variants.
- C. Additional ablation studies: The ablation compares spatially varying convolutions and positional encoding across multiple patch-grid configurations.The 1 × 1 variant evaluates the entire image as a single patch, while 16 × 16 is the original PASCAL VOC configuration.
- C. Additional ablation studies: Larger patch grids produce better accuracy in the reported ablation results.
- C. Additional ablation studies: The 1 × 1 variant is fastest because it removes network h and avoids unoptimized DPWConv operations.
- C. Additional ablation studies: The Cityscapes ablation varies the group number gwi in HyperSeg-M while tracking parameter and FLOP changes.As the number of groups increases, the parameters and FLOPs of wi decrease according to the cited equations.
D. Open source repositories
The paper lists open-source repositories used to obtain comparison information for previous methods and describes the evaluation protocol elsewhere.
- D. Open source repositories: Table 8 lists the open-source repositories used to compare against previous methods when required information was otherwise unavailable.
- D. Open source repositories: The FPS, GFLOPs, and trainable-parameter computation protocol is described in Section 4.
E. Convolution and batch normalization fusion
The paper fuses convolution and batch normalization into equivalent affine operations, including for dynamic patch-wise convolutions, to improve inference runtime.
- E. Convolution and batch normalization fusion: Inference-time fusion is applied to regular convolutions and to batch normalization following DPWConv.
- E. Convolution and batch normalization fusion: Batch normalization is represented as the affine transformation θBN · x + bBN.The passage defines the operation as matrix multiplication followed by bias addition.
- E. Convolution and batch normalization fusion: Convolution is rewritten as matrix multiplication by reshaping its weights and extracting each input feature neighborhood.The reshaped weights are θ∗ and the local input neighborhood is ˜xi,j.
- E. Convolution and batch normalization fusion: For DPWConv, batch normalization follows the patch-specific convolution and can be fused into the weight-mapping layer.The adjusted parameters are ˜θw = θBN · θw and ˜bw = θBN · bw + bBN.
F. Additional qualitative results
Additional qualitative evaluations show predictions across PASCAL VOC 2012 and CamVid, alongside representative failure cases.
- F. Additional qualitative results: PASCAL VOC qualitative results compare input images, model predictions, and ground truth across different object classes.The reported failure cases include boat–chair, missed bottles, dog–cat, and sheep–dog confusions.
- F. Additional qualitative results: CamVid qualitative results compare input, prediction, and ground-truth columns across different scenes.Reported failures include partly labeling a bicyclist as a pedestrian and missing a sign.