PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory Bandwidth, Not FLOPs: Profiling and Accelerating Query-Based Segmentation Decoders on Edge GPUs

Oliver Busemann, Torben Töniges, Martin Atzmueller

arXiv:2610.22377v1cs.ARcs.LGcs.PFeess.IV

TL;DR

Query-based segmentation decoders face memory-bandwidth limits on edge devices, where activation traffic can dominate despite standard FLOP analysis. CompactInst reduces decoder tokens with spatial pooling and restores detail with a refinement head. It speeds inference on Jetson Orin Nano while closely matching accuracy, and heavier decoder variants can still be faster when they move fewer bytes.

  • Problem

    On edge devices, transformer decoders in query-based segmentation models are memory-bandwidth-bound, with activation traffic consuming substantial decoder time and DRAM bytes.

    Source: §6 · p. 14

  • Method

    CompactInst spatially max-pools pixel features to reduce decoder tokens, then uses a refinement head to fuse decoded features with high-resolution features.

    Source: §1 · p. 2, §4 · p. 7

  • Results

    CompactInst achieves 4.74× decoder-only and 2.62× model speedups on Jetson Orin Nano at Cityscapes while reporting −0.4 AP on COCO val2017 and +0.2 AP on Cityscapes test.

    Source: §1 · p. 3

  • Takeaways & Limitations

    A variant with 2.57× more decoder FLOPs still runs 1.74× faster, supporting decoder bytes rather than FLOPs as the latency-governing factor in the tested setting.

    Source: §1 · pp. 2–3

    Uniform pooling has a small-object cost observed on COCO, and post-processing remains bandwidth-bound at high resolution.

    Source: §6 · p. 14

Abstract

from arXiv · show

Query-based segmentation models achieve strong performance even under real-time inference on workstation GPUs. On edge devices, however, we show their transformer decoders are deeply memory-bandwidth-bound, a bottleneck invisible to standard FLOP analysis that worsens with input resolution and limited DRAM bandwidth. An FP16 roofline analysis with tensor-core GEMMs active and hardware DRAM counters shows that the non-tensor-core aggregates of FastInst and Mask2Former reach only $2.5\%$ to $6\%$ of the CUDA-core ridge yet account for $53\%$ to $70\%$ of decoder time and $61\%$ to $71\%$ of decoder bytes. The largest individual kernels sustain up to $90\%$ of nominal peak DRAM bandwidth, effectively saturating the memory bus. The bottleneck is therefore not primarily low GEMM intensity, but cross-kernel activation traffic. CompactInst addresses this bottleneck using parameter-free spatial pooling at $4{\times}$, which reduces the bytes moved within the transformer decoder, while a two-convolution refinement head recovers spatial detail. Under TensorRT FP16 on Jetson Orin Nano, CompactInst delivers decoder speedups from $2.35{\times}$ to $4.74{\times}$ and model speedups from $1.57{\times}$ to $2.62{\times}$ as input resolution grows, while closely matching accuracy ($-0.4$ AP on COCO val2017, $+0.2$ AP on Cityscapes test). A heavier variant with $2.57{\times}$ more decoder FLOPs still runs its decoder $1.74{\times}$ faster on the most bandwidth-constrained device, confirming at the deployment level that decoder bytes, not FLOPs, govern latency.

1 Introduction

Query-based segmentation models can extend across perception tasks, but their high-resolution inputs make edge deployment challenging. The paper traces decoder latency to cross-kernel activation traffic and introduces CompactInst to reduce token count while preserving mask detail.

1 Introduction

  • Automotive datasets contain 3×–8× as many pixels as common COCO crops, raising the computational load for high-resolution edge deployments.Query-based models can also extend to panoptic segmentation or anomaly detection with minimal architectural changes.Source: §1 · p. 2
  • CUDA-only kernels in FastInst and Mask2Former reach just 2.5%–6% of the CUDA ridge yet account for 53%–70% of decoder time and 61%–71% of DRAM bytes.The largest kernels sustain up to 90% of nominal peak DRAM bandwidth, implicating cross-kernel activation traffic rather than low GEMM intensity.Source: §1 · pp. 2–3
  • FastInst processes each 1/8-resolution pixel token through every cross-attention and feed-forward layer, so activation traffic and decoder latency grow linearly with image area.The passage gives 4,800 tokens at 640×480 and over 32,000 at Full HD.Source: §1 · p. 2
  • CompactInst combines 4× token-reducing pooling with a refinement head, reaching 4.74× decoder-only and 2.62× model speedups on Jetson Orin Nano at Cityscapes while closely matching accuracy.It reports −0.4 AP on COCO val2017 and +0.2 AP on Cityscapes test.Source: §1 · pp. 2–3

2 Related Work

Related work spans task-specific convolutional segmentation, transformer query decoders, token reduction, and memory-bound vision models. The paper positions CompactInst as a decoder-level, edge-focused response to activation traffic not addressed by those prior diagnoses.

Instance segmentation

  • Convolutional instance-segmentation methods offer competitive latency and accuracy, but are task-specific and do not naturally generalise to panoptic or open-set segmentation.Examples include SOLOv2 [34], CondInst [32], and SparseInst [7].Source: §2 · p. 3

Query-based instance segmentation

  • DETR [3] introduced transformer object queries, while MaskFormer [5] and Mask2Former [6] extended query decoding to segmentation and multi-scale pixel features.FastInst [13] uses a dual-path decoder and 1/8-resolution pixel features that serve as both decoder tokens and final mask features.Source: §2 · p. 3

Token efficiency in vision transformers

  • Prior token-reduction work targets ViT backbones, pixel decoders, or attention, whereas CompactInst applies parameter-free max-pooling inside a lightweight segmentation decoder and restores detail with refinement.MOBIUS [30], PEM [4], and SparseRefine [21] are related scale- or token-reduction approaches; CompactInst identifies the activation-traffic mechanism behind its latency gains.Source: §2 · pp. 3–4

Memory-bandwidth bottleneck of vision transformers

  • Prior studies identify memory-bound vision operations, but the paper says the analogous diagnosis for query-based segmentation decoders on edge GPUs had not been performed.FlashAttention [9] reduces movement within attention, whereas pixel-token reduction also cuts traffic from pixel-path FFN, normalisation, and residual additions.Source: §2 · p. 4

3 The Memory-Bandwidth Bottleneck

The section profiles decoder kernels using FP16 rooflines and measured DRAM traffic, finding that activation-streaming CUDA-only kernels dominate deployment latency. This diagnosis motivates reducing pixel-token count rather than focusing only on GEMM intensity or FLOPs.

Methodology

  • The study profiles the decoder in isolation under pure FP16, measures true DRAM bytes on an RTX A4000, and compares CUDA-only and TC-only kernel aggregates against separate ridges.The A4000 is used for roofline measurements because the Jetson Tegra iGPUs expose only L2-level traffic; edge behaviour is tested separately.Source: §3 · p. 5

FP16 roofline results

  • CUDA-only kernels reach just 2.5%–6% of the CUDA ridge while accounting for 53%–70% of decoder time and 61%–71% of DRAM bytes.The largest kernels sustain up to 90% of nominal peak DRAM bandwidth; TC kernels move the minority of bytes and reach 41%–57% of their ridge.Source: §3 · p. 5, Figure 1 · p. 6, Table 1 · p. 6

FP32 roofline results

  • Under FP32, GEMM-class kernels reach arithmetic intensity 52.7 (123% of the ridge), while non-GEMM kernels have aggregate AI 1.77 and account for 36.9% of decoder time.The combined AI of 21.2 obscures these distinct kernel regimes.Source: §3 · p. 5

FP16 roofline results

  • With FP16 tensor cores active, the share of decoder time above the corresponding ridge falls to 9.3%, while non-GEMM kernels rise to 53.3% of decoder time.Tensor acceleration exposes existing cross-kernel activation traffic as the dominant runtime cost rather than creating that traffic.Source: §3 · p. 6

Implication

  • Reducing pixel-token count is the architectural lever identified to shrink cross-kernel activation traffic.The section's analysis distinguishes that traffic from low GEMM intensity as the primary bottleneck.Source: §3 · pp. 5–6

4 CompactInst: a Minimal Bottleneck Exploit

CompactInst changes only the pixel-feature path around FastInst’s transformer decoder: pooling reduces decoder tokens, and a post-decoder refinement head combines the reduced features with high-resolution features. The remaining pipeline is retained, while the refinement head’s role differs in architectures without a dual-path decoder.

4 CompactInst: a Minimal Bottleneck Exploit

  • CompactInst targets cross-kernel activation traffic by changing only the pixel-feature path immediately before and after FastInst’s transformer decoder [13].The backbone, PPM-FPN pixel decoder, queries, positional embeddings, and three decoder layers are retained.Source: §4 · p. 6
  • Parameter-free adaptive max-pooling reduces pixel features before decoding; at the default stride of 16, the pixel-token count falls by 4×.The pooled features are flattened into the token sequence consumed by the decoder layers.Source: §4 · p. 7
  • After the final decoder layer, a two-convolution Refinement Head upsamples decoded features and concatenates them with high-resolution pixel-decoder features to produce mask features.It adds approximately 1.7 M parameters, about 5% of 35.9 M, and mainly adds FLOPs rather than repeated pixel-token traffic.Source: §4 · p. 7, Figure 2 · p. 7
  • For architectures without a dual-path decoder, the Refinement Head could be omitted, but Mask2Former [6] observed an approximately 1 AP drop with a 1/16 decoder and 1/4 masks.Source: §4 · p. 7

Experiments

Experiments profile TensorRT FP16 inference across three GPUs and six resolutions, then test CompactInst’s latency, accuracy, and design choices. CompactInst reduces decoder traffic and improves speed most on lower-bandwidth devices, while accuracy trade-offs depend on dataset and resolution.

5.2 Decoder-only speedup

  • 2.35×–4.74× decoder speedups on Orin Nano accompany CompactInst’s gains across devices, which generally grow as DRAM bandwidth decreases.The decoder-only speedups are 1.49×–1.80× on A4000 and 1.47×–2.09× on AGX Orin; gains also grow with resolution, most steeply on Orin Nano.Source: §5.2 · p. 9, Figure 3 · p. 9

5.3 Model-level deployment latency

  • Up to 2.62× model speedup on Orin Nano accompanies only 7% fewer FLOPs for CompactInst, with gains generally increasing with resolution.Model timings use an R-18 backbone and exclude post-processing.Source: §5.3 · p. 9, Figure 4 · p. 10

5.4 More FLOPs, less time: the M1/4 counterfactual

  • At Cityscapes resolution on Orin Nano, M1/4’s decoder runs 1.74× faster than FastInst despite 2.57× more decoder FLOPs.M1/4 is slower on A4000, and at model level it reaches 1.33× on Orin Nano despite 2.17× as many total FLOPs; it remains slower on A4000 and AGX Orin.Source: §5.4 · pp. 9–10

5.5 Where the speedup comes from: measured DRAM traffic

  • Pooling reduces CompactInst’s decoder DRAM traffic by 64%, from 5.74 to 2.09 GB, with most of the reduction in CUDA-only activations.Only 29% of FastInst’s decoder bytes lie in the attention chain potentially reachable by fusion; after pooling, that share is 19%.Source: §5.5 · p. 10, Table 2 · p. 10

5.6 Mask accuracy

  • CompactInst reaches 37.3 AP on COCO val2017, 0.4 below FastInst, while matching FastInst on Cityscapes test with +0.2 AP.On COCO, the loss is concentrated in APS (16.3 to 15.2); on Cityscapes val under the COCO protocol, AP⋆ S improves by +0.4. The small-object trade-off is dataset- and resolution-dependent.Source: §5.6 · p. 11, Table 3 · p. 11, Table 4 · p. 11

5.7 Ablations

  • Max-pooling achieves the highest AP among the tested pooling strategies, and foreground-guided selection does not improve over either pooling baseline.The gap over average-pooling widens to +0.5 at PF 1/32; the refinement head’s removal costs 2.1 AP, with drops of 2.7 AP75 and 2.9 APS.Source: §5.7 · pp. 11–12, Table 5 · p. 12, Table 6 · p. 12
  • Retaining 1/8 pixel features before pooling and restoring detail with refinement beats direct low-resolution decoding by +2.2 AP at 1/16 and +6.6 AP at 1/32.PF 1/16 is the adopted operating point; PF 1/32 removes fewer additional tokens and costs 1.4 AP on COCO and 0.8 AP on Cityscapes test.Source: §5.7 · p. 12, Figure 5 · p. 13
  • On Cityscapes panoptic segmentation, CompactInst reaches 52.0 PQ versus FastInst’s 52.6, while its instance APTh is 23.1 versus 22.7.In matched single-path Mask2Former tests, pooling plus refinement preserves test AP at 21.9 while cutting decoder latency by 61%; PEM reduces decoder traffic less and does not lower deployment latency.Source: §5.7 · pp. 13–14, Table 7 · p. 13, Table 8 · p. 14

6 Conclusion

Under TensorRT FP16 with tensor cores, query-based segmentation decoders are limited by memory bandwidth: cross-kernel activation traffic consumes substantial decoder time and bytes, and CompactInst reduces decoder bytes while improving deployment latency. Uniform pooling remains a simple byte-reduction strategy, while high-resolution post-processing and COCO small-object costs motivate further work.

Conclusion

  • 2.62× model-level speedup on Orin Nano follows removal of 64% of decoder DRAM bytes, with matched Cityscapes AP and ∼1% relative PQ loss on panoptic segmentation.In a matched single-path Mask2Former ablation, the recipe cuts decoder latency by 61% at unchanged test AP.Source: §6 · p. 14
  • CUDA-only aggregates reach only 2.5% to 6% of the CUDA ridge yet account for 53% to 70% of decoder time and 61% to 71% of decoder bytes.The activation-streaming kernels can sustain up to 90% of nominal peak DRAM bandwidth; M1/4 has 2.57× more decoder FLOPs than FastInst yet runs 1.74× faster on Orin Nano.Source: §6 · p. 14

Limitations and future work

  • Post-processing remains bandwidth-bound, and deployment-efficient high-resolution post-processing is an open direction for future research.The authors also identify content-aware token reduction targeting bytes moved as a next step to address the COCO small-object cost.Source: §6 · p. 14

A Additional Rooflines

The appendix reports per-kernel rooflines for CompactInst and Mask2Former using the main paper’s RTX A4000 hardware-counter protocol at Full HD. Under FP16, both decoders separate low-intensity CUDA-only activation kernels from compute-dense GEMM or convolution clusters in other precision regimes.

Additional Rooflines

CompactInst

  • Under FP16, CompactInst’s CUDA-only activation kernels remain bandwidth-bound, while refinement-head convolutions form a compute-dense tensor-core cluster; FP32 shows a mixed regime.Figure A.1 plots the decoder roofline, with orange for tensor-core kernels and blue for CUDA-only kernels.Source: A Additional Rooflines · p. 18, Figure A.1 · p. 18

Mask2Former

  • Mask2Former shows the same separation: low-intensity CUDA-only activation kernels under FP16 and a compute-bound GEMM cluster under FP32.Figure A.2 presents the Mask2Former decoder roofline using the same protocol.Source: A Additional Rooflines · p. 18, Figure A.2 · p. 18

B M1/4 Deployment Latency

The M 1/4 PF 1/16 variant keeps CompactInst’s pooled transformer path but refines mask features at 1/4 resolution, increasing its arithmetic cost. On Orin Nano, it becomes faster than FastInst at sufficiently high resolutions, although the model-level crossover occurs later than the decoder crossover.

M1/4 Deployment Latency

  • At Cityscapes resolution, M 1/4 PF 1/16 has 264.9 G decoder FLOPs, 2.57× FastInst’s 103.2 G, and 446.5 G total model FLOPs, 2.17× FastInst’s 205.8 G.It retains the pooled 1/16 transformer path while refining mask features at 1/4 resolution.Source: B M1/4 Deployment Latency · p. 19
  • 1.74× decoder speedup on Orin Nano at Cityscapes resolution, while the model-level ratio reaches 1.33× only at 1024².The measured decoder ratio exceeds 1× from 640² onward; the model ratio first exceeds 1× at 1024². The variant remains slower on the A4000 and AGX Orin.Source: B M1/4 Deployment Latency · p. 19, Figure B.3 · p. 19, Figure B.4 · p. 19
  • The model-level crossover occurs later because M 1/4 upsamples the FPN from 1/8 to 1/4, adding compute outside the pooled transformer path.Figures B.3 and B.4 show decoder-only and model-level TensorRT FP16 latency across three GPUs and six resolutions.Source: B M1/4 Deployment Latency · p. 19, Figure B.3 · p. 19, Figure B.4 · p. 19

C Qualitative Comparison on Cityscapes

The qualitative comparison shows predictions from FastInst and CompactInst variants on two Cityscapes test images, with masks colored by predicted class. The M 1/4 variant improves small-instance predictions in the examples, while the most aggressive PF 1/32 reduction visibly degrades them.

Qualitative Comparison on Cityscapes

D TensorRT Reproducibility

Latency measurements use batch-1 static TensorRT FP16 engines with resolution-specific profiles and controlled timing procedures. Decoder-only timing isolates engine execution from allocation, transfers, loading, and post-processing.

D TensorRT Reproducibility

  • All latency measurements use batch-1 static TensorRT engines built from ONNX at opset 17 with FP16 and no custom plugins.Each resolution has a separate optimization profile with min = opt = max.Source: D TensorRT Reproducibility · p. 24
  • For ONNX export, adaptive max pooling is replaced by fixed max pooling with kernel and stride equal to the known pooling factor.Source: D TensorRT Reproducibility · p. 24
  • TensorRT versions and device settings vary by platform, with A4000 stock power limits, maximum-performance AGX Orin clocks, and a fixed-clock 15 W Orin Nano setup.Measurements use medians of five trials; the A4000 uses alternating trials with 50 warmup and 200 timed iterations.Source: D TensorRT Reproducibility · p. 24
  • Decoder-only timing uses preallocated synthetic feature tensors and CUDA events, excluding allocation, data transfer, engine loading, and post-processing.Source: D TensorRT Reproducibility · p. 24

E ResNet-18 Cityscapes Accuracy

ResNet-18 Cityscapes accuracy is reported as a consistency check for the models used in latency measurements. On AGX Orin, post-processing takes approximately 85 ms at Cityscapes resolution and grows with pixel count while moving full-resolution masks.

E ResNet-18 Cityscapes Accuracy

  • The main paper reports accuracy with ResNet-50 but uses ResNet-18 for model-level latency because high-resolution ResNet-50 TensorRT engines exceed Orin Nano memory.Table E.1 reports Cityscapes accuracy for the corresponding ResNet-18 models.Source: E ResNet-18 Cityscapes Accuracy · p. 24, Table E.1 · p. 24

F Post-Processing Latency

  • Approximately 85 ms of post-processing at Cityscapes resolution makes it the second-largest FastInst component and the largest CompactInst component for either backbone.The component breakdown covers 19 resolutions and separately times backbone, pixel decoder, transformer decoder, and post-processing.Source: F Post-Processing Latency · p. 25, Figure F.9 · p. 25, Figure F.10 · p. 25
  • Post-processing upsamples masks to output resolution, then thresholds and scores them, moving full-resolution tensors with little arithmetic.Its approximately linear growth with pixel count is consistent with a separate memory-bandwidth bottleneck outside the transformer decoder targeted by CompactInst.Source: F Post-Processing Latency · p. 25

Related papers

Loading 2610.22377v1…