Hardware-Attributed Operator Profiling for PyTorch

Logan Chu; Dong Li

Source-linked AI summary

Hardware-Attributed Operator Profiling for PyTorch

Logan Chu, Dong Li

arXiv:2609.11938v1Jul 4, 2026cs.ARcs.DCcs.PF

TL;DR

Operator Profiler addresses the gap between operator timing and hardware counters by matching ncu metrics to nsys kernels and attributing them to PyTorch operators through complementary paths. It achieves high coverage for compiled workloads and supports counter-informed optimization, while exposing limited coverage and evaluation scope for black-box backends and broader hardware settings.

Problem
Framework profilers expose operator timing without hardware counters, while GPU profilers expose hardware counters without operator attribution, requiring a missing bridge across incompatible tools.
Method
Operator Profiler matches ncu counters to nsys kernels by invocation order and assigns operators through CUPTI correlation, NVTX enclosure, and Inductor fusion enrichment.
Results
Across the compiled optimization case studies, attributed profiles yielded 1.76×–2.24× profiled-kernel-time speedups, while cuDNN-backed operators were surfaced as substantially unattributed.
Takeaways & Limitations
Attributed profiles provide hardware evidence for identifying bottlenecks and selecting structural or graph-level optimization actions within the evaluated workflows.
Takeaways & Limitations
The evaluation covers three workloads on one GPU architecture, uses point estimates from two measured iterations, and reports degraded attribution for cuDNN-backed operators.

Abstract

from arXiv · show

Framework profilers expose operator timing without hardware counters; GPU profilers expose hardware counters without operator attribution. Bridging this gap manually is error-prone and does not scale. We present Operator Profiler, a hardware attribution pipeline that automatically links hardware metrics to PyTorch operators via three complementary attribution paths: torch.profiler CUPTI correlation, NVTX temporal enclosure with per-stream interval trees, and Inductor fusion- map enrichment from debug artifacts. NVIDIA Nsight Compute (ncu) hardware counters are matched to NVIDIA Nsight Systems (nsys) kernel records via invocation-order matching, avoiding timestamp joins across incompatible clock domains. A curated 20-counter metric set with duration-weighted aggregation covers all hardware bottleneck axes, layer deduplication reduces ncu replay time by a factor of N/K for models with N layers across K unique structural classes, and GPU clock locking controls the kernel-duration aggregates used for operator-level comparison. On an NVIDIA RTX PRO 6000 Blackwell, Operator Profiler attributes 95-100% of kernel runtime for compiled workloads (GPT-2, SDPA Attention); black-box library backends such as cuDNN RNN are correctly surfaced as greater than 85% unattributed rather than silently dropped. Applied to profile-guided FX graph optimization, attributed profiles yield 1.76x-2.24x profiled-kernel-time speedups on the two compiled optimization case studies; a third LSTM diagnostic case identifies cuDNN re-dispatch as a structural fix rather than an FX graph rewrite.

1 Introduction

Operator Profiler addresses the missing bridge between PyTorch operator timing and GPU hardware counters. It combines cross-tool counter matching with complementary operator-attribution paths and supports hardware-informed optimization.

Motivation: Framework profilers expose operator timing without hardware counters, while GPU profilers expose counters without operator attribution across incompatible collection passes.The resulting gap makes operator-level hardware diagnosis difficult and manual bridging non-scalable.
PyTorch compilation stack: PyTorch compilation fuses operators into Triton kernels, so individual aten:: dispatches may not occur at runtime.Vendor-library extern calls can preserve per-operator dispatch, whereas fused operations may represent multiple operators in one kernel.
Contributions: Operator Profiler joins ncu hardware counters to nsys kernels by invocation order, then assigns operators through CUPTI correlation, NVTX enclosure, and Inductor fusion enrichment.The pipeline separates hardware-counter matching from operator identity assignment.
System overview: The system overview separates profiling, attribution into profile.json, and downstream optimization, with optimization outside the attribution pipeline.The profiling stage collects kernel timelines and hardware counters before the attribution engine combines them.

2 Related Work

Existing tools separately provide operator timing, kernel timelines, hardware counters, or compilation optimization, but do not close the attribution loop. Operator Profiler combines these capabilities to support counter-informed FX optimization.

Operator-level profilers: torch.profiler provides operator timing and CUPTI correlation but no hardware counters, leaving hardware-resource contention and optimization choices unresolved.Its timing-centric view identifies slow operators without identifying the resource they contend for.
Kernel-level profilers: Nsight Systems captures kernel timelines and Nsight Compute collects per-kernel counters, yet neither exposes the operator graph needed for kernel-to-operator attribution.Cluster monitoring tools operate at a coarser granularity and do not supply this attribution.
Deep learning compilers and optimizers: TorchInductor and related compilers make fusion decisions without hardware-counter feedback about utilization changes.Black-box runtimes such as ONNX Runtime and TensorRT also cannot readily be extended with custom attribution data.
Fused kernel libraries: Fused attention libraries motivate hardware-counter-guided selection rather than unconditional use, but lack a profiling layer that identifies the warranted variant per workload.The missing capability is measured per-operator evidence tied to optimization choice.
Profile-guided optimization: Operator Profiler combines nsys traces, application-mode ncu counters, and PyTorch attribution through invocation-order matching to enable hardware-informed FX optimization.The paper presents this combination as a first published system of its kind.

3 Attribution Pipeline

The attribution pipeline resolves non-unique kernel names, incompatible clocks, and missing operator identity by combining counter matching with prioritized attribution paths. It also controls replay cost and duration comparability for compiled workloads.

3 Attribution Pipeline: The pipeline ingests nsys kernel activity and NVTX annotations plus ncu per-kernel counters, producing an OperatorProfile that maps operators to kernels, metrics, and confidence.Fused kernels can be annotated with all constituent aten:: operators.
3.1 Data Ingestion: Application-mode ncu replay collects all 20 metrics in 4–8 counter-group passes, reducing cost to O(counter groups) and achieving a 10–50× reduction versus unique-kernel replay.This is the reliable collection mode for Inductor-compiled workloads.
3.2 GPU Clock Locking for Reproducible Duration Comparisons: Kernel durations come from nsys rather than ncu, so GPU clock variation can bias baseline-versus-optimized duration ratios without clock locking.The confound includes boost-clock changes, thermal throttling, and different power dissipation across captures.
3.2 GPU Clock Locking for Reproducible Duration Comparisons: Probe-and-lock selects a sustained frequency and reuses it across baseline and optimized nsys captures, making their duration ratio clock-immune by construction.Clock locking applies to nsys capture only; ncu replay independently self-locks its base clock.
3.3 The Clock Domain Problem and Invocation-Order Matching: Invocation-order matching replaces timestamp joins because nsys and ncu clocks diverge by 10–100 µs per kernel and accumulate error across the trace.The algorithm orders nsys kernels per stream and ncu rows by kernel name and invocation index.
3.4 torch.profiler Correlation Attribution (HIGH Confidence): CUPTI External Activity correlation provides the highest-confidence attribution by linking each operator dispatch causally to the kernels it launches.This path requires neither timestamp comparison nor pattern matching.

3.5 StreamIntervalTree Attribution

Operator Profiler attributes kernels to PyTorch operators through per-stream NVTX enclosure, with fallback enrichment from Inductor debug artifacts. The pipeline also handles replay caching, fused operators, deduplication, and hardware-metric aggregation for reliable operator-level profiles.

NVTX temporal enclosure: Per-stream interval trees support efficient NVTX enclosure queries using host-side launch timestamps, which match CPU-domain annotations to asynchronously launched CUDA kernels.GPU execution timestamps are avoided because the host-to-GPU asynchronous gap makes them unreliable for enclosure queries.
Replay and fusion handling: CUDA graph replays reuse attribution cached during graph capture, while multiple enclosing NVTX ranges mark kernels as fused across operator boundaries.Kernels without an enclosing range can remain unattributed or receive Inductor enrichment.
Inductor fusion enrichment: When NVTX enclosure fails, Inductor debug comments recover the exact constituent aten:: operators for fused kernels through a post-attribution fusion map.The enrichment pass upgrades otherwise unattributed kernels and augments valid attributions without overriding them.
Layer deduplication: Structural deduplication profiles one representative per equivalence class, reducing ncu replay from O(N) layer partitions to O(K) unique structural classes.Duplicate partitions inherit metrics from their representative, but partition tags must be emitted only during measured capture rather than warmup.
Hardware-metric aggregation: Duration-weighted aggregation gives long-running kernels proportional influence on rate and utilization metrics, while additive metrics are summed and per-kernel constants use their maximum.This prevents short auxiliary kernels from receiving equal weight with dominant compute kernels and preserves the highest-pressure occupancy constraint.

4 Evaluation: Attribution Quality and Profile-Guided Optimization

The evaluation measures attribution coverage and tests whether per-operator hardware profiles can guide and verify optimization decisions. Across compiled workloads, attribution localizes hardware bottlenecks to concrete operator or graph structures, while the LSTM case uses unattributed execution to diagnose a structural backend issue.

Attribution Coverage: GPT-2 and SDPA Attention achieved more than 95% attributed runtime, whereas LSTM’s more than 85% unattributed fraction reflected opaque cuDNN RNN dispatch.The attribution coverage result distinguishes successful compiled-workload attribution from an expected black-box backend boundary.
GPT-2: GPT-2 attribution localized 87.5% of attributed runtime to GEMMs with zero Tensor Core activity, making BF16 promotion a calculable optimization target.The GEMM family accounts for 6,401 of 7,313 µs and uses 210 registers per thread.
SDPA Attention: Q/K/V projection GEMMs held 96.7% of baseline attributed runtime at 16.6% occupancy, and their shared LayerNorm input exposed the precondition for QKV fusion.The profile-to-graph chain connected the counter symptom to a specific FX transformation.
SDPA Attention: QKV fusion raised achieved occupancy from 16.6% to 73.9%, while auxiliary kernels accounted for 9.1% of optimized runtime and bounded the observed 2.24× speedup.BF16 promotion and Inductor freezing were applied alongside QKV fusion.
LSTM Sequence Encoder: The LSTM diagnostic routed the recurrent region to a fused cuDNN Tensor-Core RNN, reducing attributed execution time by 2.26 ms while all FX passes remained unapplied.The fused launch used M=4096 instead of M=32 and eliminated all 1,280 splitKreduce epilogues.

5 Conclusion

Operator Profiler bridges hardware counters and PyTorch operators, enabling counter-informed optimization while exposing scope, attribution, and reproducibility boundaries.

5 Conclusion: Invocation-order matching and three attribution paths connect ncu counters to nsys kernels and PyTorch operators without cross-tool timestamp joins.The pipeline uses CUPTI correlation, NVTX temporal enclosure, and Inductor debug-artifact enrichment.
5 Conclusion: Across three case studies, attributed profiles support profile-guided FX optimization, while the LSTM case routes a structural dispatch problem to a structural fix.The pipeline is also positioned for neural architecture search, serving-stack profiling, and CI regression detection.
5 Conclusion: Evaluation is limited to three workloads on one GPU architecture, with point estimates from two measured iterations and no characterized end-to-end wall-clock variance.Coverage also degrades for cuDNN-backed operators, and conditional or dynamic workloads can violate invocation-order assumptions.
5 Conclusion: Future work targets multi-GPU attribution, online CUPTI Perfworks profiling, cross-architecture validation, and automated mapping from counter symptoms to graph transformations.Counter shifts may also support CI regression testing independently of an optimization framework.

A.1 Hardware Specification

The evaluation hardware is a single NVIDIA RTX PRO 6000 Blackwell system, with Blackwell-specific counter availability affecting latency-bound measurement.

A.1 Hardware Specification: All experiments run on one NVIDIA RTX PRO 6000 Blackwell GPU with approximately 188 SMs, 96 GB GDDR7, and approximately 1.8 TB/s memory bandwidth.The device uses 5th-generation Tensor Cores and Blackwell sm 120.
A.1 Hardware Specification: The Blackwell counter set lacks warp cycles per instruction, so eligible cycles percentage serves as the latency-bound indicator.
A.1 Hardware Specification: The software stack requires an NVIDIA GPU, nsys ≥2024.6, ncu ≥2025.4.1, PyTorch 2.11+, and CUDA 12.8.

A.2 Hardware Counter Reference

The counter reference documents a curated 20-counter set organized by bottleneck axis and aggregation method, with a Blackwell-specific latency indicator.

A.2 Hardware Counter Reference: Table 8 organizes 20 hardware counters by bottleneck axis, measurement, and aggregation method for multi-kernel operator attribution.The set covers the counters collected by Operator Profiler and specifies how attributed kernels are aggregated.
A.2 Hardware Counter Reference: On Blackwell sm 120, eligible cycles percentage replaces the unavailable warp cycles per instruction counter as the latency-bound indicator.

A.3 Prototype Automation via Claude Code Plugin

The prototype Claude Code plugin automates the closed-loop profiling and optimization protocol through one command and five specialized agents.

A.3 Prototype Automation via Claude Code Plugin: The plugin runs the closed-loop protocol through a single `/optimize workload.py` command.
A.3 Prototype Automation via Claude Code Plugin: Five agents—capture-agent, optimization-strategist, backend-engineer, validationagent, and /report—cover baseline capture through cross-profile comparison.

A.4 Annotated profile.json Schema

The annotated profile.json schema records operator-to-kernel attribution, execution metadata, and aggregated hardware metrics, with fields documenting fusion, confidence, and architecture-specific counter availability.

Schema overview: profile.json maps each dispatched PyTorch operator to its constituent CUDA kernels, nsys durations, and aggregated hardware metrics.Table 9 provides the top-level field reference, while the excerpt illustrates an NVTX-attributed aten::mm record.
Operator identity and fusion: Operator entries identify call order and fusion status, including the aten:: operations fused into a shared kernel.The representative record is aten::mm_26 at call index 26 and is not fused.
Kernel records: Each kernel record includes an attribution method, confidence, duration, stream identifier, launch dimensions, and raw ncu counter fields.The example records NVTX attribution with medium confidence, a 32,736 ns duration, stream 7, grid and block dimensions, and 20 architecture-specific counter fields.
Aggregated metrics: Aggregated operator records summarize duration and kernel count alongside tensor-core activity, occupancy, throughput, cache hit rate, register use, eligible cycles, and spills.The example also preserves a null warp-cycles-per-instruction value because the corresponding Blackwell counter is absent.

A.5 Supplementary Figures

The supplementary material documents composable FX pass levels and their distinct roles across the torch.compile pipeline.

Supplementary method: Passes at each FX level compose without interfering across levels.
Three-level FX pass hook points: Level 1 passes must run before AOTAutograd because decomposition can erase shared LayerNorm-to-QKV structure needed for QKV fusion.Level 2 operates on Aten IR after decomposition, while Level 3 controls Inductor behavior.