Source-linked AI summary
Distributed Semantic Segmentation With Improved Rate-Distortion Trade-Off
Danish Nazir, Timo Bartels, Thorsten Bagdonat, Tim Fingscheidt
TL;DR
Distributed semantic segmentation must transmit compressed bottleneck features from constrained edge devices while preserving segmentation quality, but zero-mean hyperprior codecs yield suboptimal low-bitrate RD trade-offs. The paper proposes HSM and its autoregressive extension AR-HSM, which model latent means and standard deviations and, for AR-HSM, channel dependencies. The methods achieve state-of-the-art performance below 0.2 bpp on ADE20K and 0.03 bpp on Cityscapes.
Problem
Existing distributed semantic segmentation methods rely on zero-mean hyperprior entropy models that inadequately capture latent distributions, limiting RD performance at low bitrates.
Method
The paper proposes HSM, which predicts latent means and standard deviations, and AR-HSM, which adds channel-wise autoregressive entropy modeling.
Results
HSM and AR-HSM achieve state-of-the-art distributed semantic segmentation below 0.2 bpp on ADE20K and 0.03 bpp on Cityscapes.
Takeaways & Limitations
AR-HSM consistently outperforms HSM and existing hyperprior baselines across investigated bitrates and encoder backbones.
Takeaways & Limitations
HSM does not explicitly capture inter-channel dependencies within the latent representation, motivating AR-HSM.
Abstract
from arXiv · showhide
Distributed deep neural networks (DNNs) for dense perception tasks such as semantic segmentation execute an encoder DNN on edge devices, and a decoder DNN typically on a large-scale cloud platform with a particular constraint on transmission bitrate. Recent works employ source codecs to enable bitrate-efficient transmission between the edge device and the cloud. However, as these approaches are typically bound to a particular type of source codec and alternative network architectures are often not explored, this results in a suboptimal rate-distortion (RD) trade-off in the low-bitrate regime. In this work, we propose two novel source codecs that \textit{enable extremely low bitrates, while improving RD performance}. We demonstrate the effectiveness of our proposed source codecs by achieving state-of-the-art performance in distributed semantic segmentation at below 0.2 (0.03) bits per pixel, measured using the mean intersection-over-union metric on ADE20K (Cityscapes).
1 Introduction
Distributed semantic segmentation addresses edge-device constraints by compressing bottleneck features for transmission, but existing hyperprior codecs assume zero-mean Gaussian latents, limiting low-bitrate RD performance. The paper proposes HSM and AR-HSM codecs that model latent means and standard deviations, with AR-HSM additionally modeling channel dependencies.
- Motivation: Distributed semantic segmentation executes image encoding on resource-constrained edge devices and decoding in the cloud.This distribution reduces the computation required on camera-enabled devices while requiring bitrate-efficient communication.
- Limitations of existing codecs: A zero-mean Gaussian entropy model can inflate predicted latent variance and produce a suboptimal RD trade-off, especially at low bitrates.The RD trade-off relates transmission bitrate to semantic segmentation performance at that bitrate.
- Proposed codecs: HSM models each latent element using both predicted mean and standard deviation instead of assuming zero mean.This design targets local context-dependent distribution shifts and reduces predicted latent variance.
- Proposed codecs: AR-HSM extends HSM with channel-wise autoregressive entropy modeling to further improve RD performance in the low-bitrate regime.The extension addresses dependencies that HSM does not explicitly capture.
- Reported outcome: HSM and AR-HSM achieve state-of-the-art distributed semantic segmentation below 0.2 bpp on ADE20K and 0.03 bpp on Cityscapes.Performance is measured using mean intersection-over-union.
2 Related Works
Semantic segmentation assigns every image pixel to a semantic category, with transformer-based architectures becoming prominent through their ability to model long-range dependencies. Distributed execution and low-complexity source codecs address edge-device constraints and bitrate efficiency, while HSM removes the zero-mean assumption and motivates AR-HSM to model channel dependencies.
- Semantic segmentation: Semantic segmentation is a dense prediction task that assigns each input-image pixel to a predefined semantic category.Transformer-based methods have become prominent because they efficiently model long-range dependencies.
- Distributed semantic segmentation: Distributed semantic segmentation splits execution across client and server to address camera-enabled edge devices’ computational and power constraints.The distributed paradigm moves part of the semantic segmentation DNN away from the edge device.
- Source codecs: Low-complexity source codecs compress latent bottleneck features to improve bitrate efficiency, following learned image-compression approaches.Earlier factorized-prior designs exhibit inferior RD performance compared with later approaches.
- HSM and AR-HSM: HSM avoids the zero-mean assumption because it can inflate predicted variance and worsen the RD trade-off at low bitrates.The proposed codec models the latent representation’s mean and standard deviation.
- HSM and AR-HSM: AR-HSM is introduced to exploit inter-channel dependencies that HSM does not explicitly model.This extension is intended to further improve RD performance for distributed semantic segmentation.
3 Methods
The framework compresses bottleneck features between an edge-side encoder and cloud-side decoder using hyperprior source codecs. HSM models latent means and standard deviations, while AR-HSM additionally models channel-wise dependencies for lower-bitrate transmission.
- Overview: The distributed segmentation pipeline uses an edge-side feature encoder and compression encoder, then reconstructs bottleneck features in the cloud for task decoding.The compression encoder produces latent and enhancement bitstreams; the cloud compression decoder reconstructs the latent representation used to generate the segmentation map.
- HSM source codec: HSM uses a hyperprior architecture to predict both standard deviation and mean parameters for entropy modeling of latent bottleneck features.Its compression encoder quantizes the latent representation and produces latent and enhancement bitstreams, while the hyperprior models spatial dependencies.
- HSM source codec: The proposed HSM decoder adds a hyperprior mean decoder alongside the standard deviation decoder, removing the baseline zero-mean Gaussian assumption.The added mean decoder slightly increases computational complexity and learnable parameters, while improving entropy modeling of the latent representation.
- AR-HSM source codec: AR-HSM partitions latent channels and applies channel-wise autoregressive entropy modeling to capture inter-channel dependencies.The architecture uses two equally sized channel partitions and refinement blocks for standard deviation and mean parameters.
- AR-HSM source codec: AR-HSM uses a low-complexity hyperprior decoder rather than the mixed CNN-transformer decoder used in the referenced learned image compression codec.This design targets distributed semantic segmentation while retaining autoregressive modeling of latent dependencies.
4 Experimental Setup
Experiments evaluate the proposed source codecs on ADE20K and Cityscapes using common training objectives, multiple baselines, and rate-distortion metrics based on bitrate and mIoU.
- Datasets: ADE20K and Cityscapes provide the experimental datasets, covering diverse annotated scenes and urban-scene understanding, respectively.ADE20K contains 150 classes, while Cityscapes contains 19 semantic classes with fine-grained pixel-level annotations.
- Baselines: Four baselines are evaluated: no compression, HS, inference-time AR-HS (μ=0), and reference AR-HS without mean prediction.The same experiments use CNNs and transformers on ADE20K and Cityscapes.
- Training objective: The training rate objective uses expected negative log probabilities for the latent representation and hyperprior representation.The objective averages over training minibatches and uses discrete distributions P̂r and P̂h.
- Training objective: The distortion objective uses one-hot ground-truth semantic labels whose class probabilities sum to one at each pixel.The rate and distortion terms are combined into the total loss with hyperparameter α∈(0,1), using the same objective across HS, AR-HS, HSM, and AR-HSM.
- Metrics: RD performance is measured in bits per pixel for rate and mIoU for distortion, with adapted BD-rate quantifying average bitrate savings between RD curves at identical mIoU.The mIoU-adapted BD-rate comparison is illustrated in Figure 9.
5 Experimental Results and Discussion
Experiments show that HSM consistently improves the rate-distortion trade-off over HS, while AR-HSM performs best across datasets, backbones, and bitrates. The proposed codecs achieve these gains with little additional computational or parameter cost, although backbone selection creates practical trade-offs between BD-rate, complexity, and mIoU.
- Rate-distortion performance: HSM exceeds the HS baseline across investigated bitrates and encoder backbones on ADE20K, while AR-HSM performs even better at all bitrates.The same observations are reported for Cityscapes, with the results suggesting generalization across datasets.
- Rate-distortion performance: AR-HSM’s zero-mean ablation produces much larger bitrates, supporting the effectiveness of predicting the latent mean during inference.The ablation is evaluated on both MiT-Bn and RN-50 encoders.
- BD-rate savings: HSM consistently improves BD-rate over HS for all encoder backbones and both datasets, while AR-HSM exceeds HSM and establishes the reported SOTA.Table 2 reports separate results for RN-50 and MiT-B0 through MiT-B5.
- Complexity: HSM and AR-HSM have similar GFLOPs and parameter demands to the corresponding encoder backbones, indicating little extra resource requirement.Figure 11 compares BD-rate savings against total GFLOPs and parameter count.
- Complexity: For MiT-B1 on ADE20K, HSM reaches −45.5% BD-rate at 14.3 GFLOPS versus AR-HSM’s −40.1% at 24.4 GFLOPS, illustrating a complexity–performance trade-off.The text notes that HSM’s lower complexity comes with some mIoU degradation.
- Dataset comparison: On Cityscapes, AR-HSM achieves BD-rate below −60% for all investigated backbones, while AR-HS produces higher bitrates for three of six backbones.Cityscapes uses the higher image resolution setting described in the experiments.
6 Conclusions
The paper addresses suboptimal low-bitrate RD performance caused by zero-mean hyperprior codecs in distributed semantic segmentation. It proposes HSM and its autoregressive extension AR-HSM, which improve latent modeling and outperform prior SOTA methods on ADE20K and Cityscapes across backbones and model sizes.
- The paper targets improved rate-distortion performance in distributed semantic segmentation, particularly in the low-bitrate regime.
- HSM predicts both the mean and standard deviation of latent representations, while AR-HSM additionally models channel-wise autoregressive dependencies.
- HSM and AR-HSM exceed prior SOTA on ADE20K and Cityscapes across various backbones and model sizes.