Source-linked AI summary
Accelerated Decoding of Centroid Positional Encoding for Instance Segmentation
Carmelo Scribano, Filippo Muzzini, Nedyalko Prisadnikov, Mohammad Mahdi, Yuqian Fu, Giorgia Franchini, Danda Pani Paudel, Marko Bertogna, Luc Van Gool
TL;DR
Prediction decoding is an underexplored source of inference cost, particularly for sinusoidal centroid encoding in instance segmentation. The paper develops a CUDA decoder that reorganizes voting, peak detection, and mask aggregation for GPU execution, reducing decoding and end-to-end latency. The optimized implementation reaches speedups of up to 55% in INT8 mode at 280 × 280 resolution and outperforms reference implementations across reported scaling tests.
Problem
Prediction decoding can impose substantial overhead because generic CPU routines and inefficient GPU kernels limit the benefits of efficient vision models.
Method
The paper develops a hardware-aware CUDA decoder for sinusoidal centroid encoding, optimizing histogram voting, local-maxima search, and mask aggregation.
Results
Up to 55% speedup in INT8 mode at 280 × 280 resolution, with the decoder consistently outperforming the PyTorch reference and scaling better as feature resolution increases.
Takeaways & Limitations
Efficient decoding is necessary to realize the practical and real-time benefits of advanced output representations.
Takeaways & Limitations
Memory management accounts for 33% of the remaining latency and remains a target for future custom low-level optimization.
Abstract
from arXiv · showhide
Beyond model inference, the decoding stage, which converts raw network outputs into task-level representations, constitutes a significant portion of the execution cost. Despite its practical impact, prediction decoding has received comparatively little attention and is often implemented using generic CPU routines or inefficient GPU kernels, limiting the benefits of advances in model efficiency. In this work, we investigate the decoding overhead associated with a recent sinusoidal centroid encoding for Instance Segmentation, in which each pixel regresses a positional embedding of its instance centroid. This approach allows flexible segmentation without predefined proposals, but extracting instance masks from dense embeddings incurs a high computational cost. We present an optimized CUDA-based implementation of the decoding algorithm tailored to this encoding, explicitly addressing challenges related to parallelization, synchronization, and memory access on modern GPUs. Our solution significantly reduces decoding overhead and improves End-to-End inference latency, outperforming both CPU-based approaches and naive GPU implementations. The results demonstrate that efficient decoding is essential to fully exploit the advantages of advanced output representations and highlight the importance of jointly designing encoding schemes and their decoding algorithms for real-time computer vision systems.
1 Introduction
Prediction decoding can impose substantial overhead despite advances in model architecture and compression, especially in resource-constrained real-time deployments. This work targets sinusoidal centroid encoding for instance segmentation with a CUDA decoder designed to reduce that bottleneck.
- Motivation: Decoding can significantly increase end-to-end cost and may dominate inference because generic CPU routines and inefficient GPU kernels cause memory-bound operations, synchronization, and poor accelerator utilization.These issues are especially relevant for edge devices requiring low-latency processing.
- Motivation: Sinusoidal centroid encoding supports an arbitrary number of instances without complex proposal logic, but recovering masks remains a bottleneck because decoding is multi-stage.
- Contribution: The proposed high-performance CUDA implementation addresses parallelization and memory-access challenges in the decoder for practical real-time deployment.
- Contribution: The study argues that the practical value of novel output representations depends closely on the efficiency of their associated decoding algorithms.
2 Background and Related Work
Instance segmentation assigns pixels to object instances with an unknown count, while related segmentation tasks assign semantic labels or combine instance and semantic outputs. The paper studies a bottom-up centroid-encoding approach and its GPU decoding context.
- Instance Segmentation: The paper focuses on class-agnostic instance segmentation, distinct from semantic segmentation and panoptic segmentation.
- Instance Segmentation: Instance segmentation must localize objects and assign pixels to a variable number of instance masks because the number of instances is unknown in advance.Top-down methods use proposals, whereas bottom-up methods aggregate dense pixel-level embeddings after inference.
- Decoding Overview: Figure 1 presents an overview of the proposed encoding's decoding process.
- Related Work: The studied approach was previously combined with a DINOv2 backbone and designed loss to achieve state-of-the-art COCO results, while extending dense-embedding paradigms such as Painter and DCME.
- GPU Computing: CUDA exposes GPU parallelism through grids and blocks, with shared memory and synchronization barriers enabling fine-grained resource control.
3 Methodology
The method encodes each pixel with the sinusoidal positional embedding of its instance centroid, then decodes noisy dense predictions through voting, peak detection, and mask assignment. Its CUDA implementation precomputes encoding palettes and replaces contended atomic accumulation with factorized matrix operations.
- Encoding: Each pixel encodes the sinusoidal positional embedding of its instance centroid, while background pixels use a void encoding.The encoding uses L = 4 harmonics.
- Decoding Algorithm: The decoder builds a coarse histogram by having each output location cast soft votes to candidate centroid bins using separable encoded-space distances and exponential vote weights.Votes are aggregated over participating output locations.
- Decoding Algorithm: Local-maxima search extracts candidate centroids by suppressing weak noisy peaks, after which each output pixel is assigned to the closest active centroid satisfying the affinity threshold.
- Decoding Implementation: The CUDA implementation precomputes x- and y-direction palettes for evenly spaced bin coordinates and stores them as model parameters to avoid recomputation and memory copying.Background pixels are implicitly handled without an explicit void encoding, leaving Bw × Bh histogram bins.
- Decoding Implementation: To avoid global atomic contention during histogram accumulation, the implementation factorizes vote weights into Ex and Ey matrices and computes the histogram with cublasSgemmStrided.Specialized kernels independently compute the factorized matrices before matrix multiplication.
4 Results and Discussion
The evaluation shows that optimized CUDA decoding improves latency and stability while preserving task-quality trade-offs, with benefits extending from isolated decoder execution to end-to-end inference.
- Experimental setup: Latency, memory footprint, and PQ are evaluated on an NVIDIA Jetson Orin Nano using TensorRT, with decoder timing measured in isolation through a minimal ONNX graph.Data-transfer latencies are omitted, and custom CUDA allocation tracks peak memory usage.
- Impact of the number of bins: PQ gains plateau at 32 × 32 histogram bins, so higher resolutions add latency without significant quality improvements.The 32 × 32 configuration is adopted for subsequent evaluations.
- Impact of the feature resolution: The proposed decoder consistently outperforms the PyTorch reference and scales better as feature-map resolution increases with fixed 32 × 32 bins.The optimized kernels limit the additional computational cost associated with higher-resolution inputs.
- End-to-End Latency Breakdown: Up to 55% lower End-to-End latency is achieved in INT8 mode at 280 × 280 when replacing the PyTorch decoder with the proposed CUDA implementation.Both pipelines use the same TensorRT-optimized backbone and intermediate components, isolating the decoding stage as the difference.
- Kernel Execution Profiling: Histogram accumulation contributes most to decoder latency, followed by mask aggregation, while cublasSgemmStrided and local maxima search have marginal impact.Memory operations account for 33% of total decoder inference time after raw GPU compute is reduced.
- Kernel Execution Profiling: Decomposing H, using cuBLAS, and optimizing mask aggregation drive the improvement over the naive implementation, including an approximately 6fold mask-aggregation speedup.The naive and optimized kernels are mutually incompatible because their preceding pipeline stages use different data encodings.
5 Conclusion
The work introduces a high-performance GPU-accelerated decoder for instance segmentation that achieves substantial speedups over existing implementations. It also identifies memory management as a remaining latency source for future optimization.
- The GPU-accelerated decoder achieves substantial speedups over existing implementations.
- Optimizing traditionally CPU-bound auxiliary stages is as critical to reducing End-to-End latency as refining the core model architecture.
- Memory management accounts for 33% of the remaining latency and is targeted for future optimization through custom low-level orchestration.