PaperlayerView exact revision on arXiv

Source-linked AI summary

Xtrace: High-Fidelity GPU Intra-Kernel Tracing via Binary-Level Instruction Splicing

Zhuobin Huang, Kai Zhang, Weihao Cui, Hongshi Tan, Liang Luo, Christopher Dewan, Shen Li, Bingsheng He

arXiv:2609.28769v1cs.AR

TL;DR

Fused GPU kernels need fine-grained timing, but pre-compilation probes can change the binary and add runtime overhead. Xtrace splices probes into compiled binaries; on production LLM kernels, it preserves more instructions with less overhead than existing tools.

  • Problem

    Pre-compilation tracing can interfere with compiler optimizations and produce a binary different from the one intended for observation, while probes also add execution-time overhead.

    Source: §1 · p. 2

  • Method

    Xtrace splices probes into compiled GPU binaries, reusing dead registers and resolving hazards to minimize interference with the kernel.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    Xtrace preserves 94–98% of kernel instructions and adds 0.9–2.8% overhead, versus 8–48% preserved and 3.8–75.6% overhead for existing tools.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Xtrace-guided optimization reaches the same FA-3 performance in 3.9× fewer iterations than the best existing tracing tool and supports trace-guided optimization against closed-source cuDNN.

    Source: §1 · p. 2, §5.2.1 · p. 10, §6 · p. 13

    IKET cannot run with profiling tools, so its kernel duration cannot be measured directly; the evaluation therefore uses a bit-identical reimplementation.

    Source: §5.4.1 · p. 12

Abstract

from arXiv · show

Modern GPU kernels fuse increasingly more work into a single kernel, and intra-kernel tracing has become the mainstream method to profile them. Tracing inserts probes into the kernel to record its runtime states, and the fidelity of the trace determines the efficiency of performance optimization. Unfortunately, existing tools insert probes before compilation. These tools interfere with the compiler's optimizations, so they trace a different binary from the one the GPU executes. They also add significant runtime overhead. Xtrace is the first GPU kernel tracing system with near-zero compile-time interference and minimized runtime overhead. Xtrace inserts probes directly into the compiled kernel binary. It reuses only the registers that hold dead values at the insertion address and resolves all hazards with the compiler's hazard tables. It further schedules the instruction order, register allocation, and control bits to minimize the runtime overhead the probe introduces. Xtrace supports 19 NVIDIA and AMD GPU architectures, and is publicly available for use at https://g-watch.github.io. We evaluate Xtrace on major production large language model (LLM) kernels against the state-of-the-art tracers Neutrino and IKET from NVIDIA. On H100, B300, and MI300X GPUs, Xtrace preserves 94-98% of the instructions of the kernel, while existing tools preserve only 8-48%. Xtrace adds only 0.9-2.8% overhead, while existing tools add 3.8-75.6%. Xtrace guides a coding agent to reach the same FlashAttention-3 performance with 3.9x fewer iterations than existing traces do. Thanks to our binary-level instrumentation, Xtrace also traces the faster closed-source cuDNN kernel, which guides the agent to lift the open-source FlashAttention-4 by 5.2-13.3% in throughput.

1 Introduction

Fine-grained tracing can expose execution behavior in fused GPU kernels, but pre-compilation probes alter the binary and add runtime overhead. Xtrace splices probes into compiled binaries to reduce these distortions.

1 Introduction

  • Intra-kernel tracing exposes fine-grained runtime states, but is useful only when it preserves original instruction behavior and reveals the probed kernel’s true execution time.Accurate timing helps expose instruction overlap, pipeline bubbles, and idle periods in fused workloads.Source: §1 · pp. 1–2
  • Neutrino [35] preserves only 8–20% binary similarity and IKET [56] 13–48%, while their overheads are 11.8–75.6% and 3.8–28.4%, respectively.The tools insert probes or reserve resources before compilation, changing the traced binary and competing with the kernel for registers.Source: §1 · p. 2
  • Xtrace inserts probes after compilation, reuses registers that are dead at the insertion point, resolves hazards, and schedules probes to minimize dependencies and perturbation.Binary-level instrumentation leaves compiler optimizations intact and avoids competing for registers when idle registers are available.Source: §1 · p. 2
  • Across every tracing workload, Xtrace preserves 94–98% of compiled-kernel instructions and adds only 0.9–2.8% overhead, compared with 3.8–75.6% overhead for existing tools.It leaves every basic block intact and adds no registers to the kernel.Source: §1 · p. 2
  • Xtrace reaches the same FA-3 performance in 3.9× fewer iterations than the best existing tracing tool and lifts FA-4 throughput by 5.2–13.3% against closed-source cuDNN.The agent-guided FA-4 optimization is enabled by tracing the closed-source vendor kernel.Source: §1 · p. 2

2 Background & Motivation

Intra-kernel tracing exposes fine-grained GPU execution, but its usefulness depends on preserving the execution being measured. Existing pre-compilation probes can alter compiler optimizations and add runtime waits, motivating binary-level insertion that reuses available resources while preserving kernel behavior.

2.1 GPU Intra-Kernel Tracing

  • Intra-kernel tracing records runtime state such as timestamps to expose fused kernels’ execution behavior beyond aggregate profiler statistics.Existing tools insert probes at source, intermediate-representation, or binary levels.Source: §2.1 · p. 3

2.2 Existing Tools Incur Significant Fidelity Loss

  • Neutrino and IKET probes increase FA-3 execution time by 13.0% and 5.3%, while probe stores and warp synchronization add waits absent from the unprobed kernel.A producer probe store waits 77% of the time; a global memory access takes 270 to 380 ns on Hopper GPUs.Source: §2.2 · pp. 3–4
  • Probes prevent compiler optimizations in FA-3, producing sequential SoftMax passes and a delayed V-tile release that inflate reported times.The traced SoftMax time reaches 992 ns, and the reported V-tile wait is 1,024–1,216 ns.Source: §2.2 · p. 3, Figure 4 · p. 4

2.3 Opportunities and Challenges

  • Binary-level instruction insertion avoids compiler interference, but requires identifying safely reusable registers, resolving hazards, and minimizing probe-induced waits.The proposed approach inserts probe instructions directly into the compiled kernel’s instruction sequence.Source: §2.3 · pp. 4–5

3 Xtrace

Xtrace splices probes into compiled GPU binaries, analyzing resources available at each insertion point and optimizing the insertion to preserve kernel behavior with minimal overhead. Its probe library, splicer, and runtime driver connect source-level trace hints to collection and parsing of runtime traces.

Overview

  • Xtrace analyzes available registers and scheduling resources at an insertion address, then derives a correct, low-overhead probe insertion called splicing.It assigns only dead registers, resolves hazards with compiler hazard tables, and minimizes estimated dependency waits.Source: §3 · p. 5

System components and workflow

  • The probe library supplies templates, the splicer inserts them at source-hint locations in the binary, and the runtime driver intercepts launches to run traced kernels.When source-to-binary mapping is unavailable, Xtrace also supports insertion at a specified binary program address.Source: §3 · p. 5, Figure 6 · p. 5
  • The runtime driver allocates a launch-sized device buffer, runs the probed kernel, then parses the collected trace.The driver reads kernel parameters to size the buffer before running the probed kernel in place of the original.Source: §3 · p. 5

4 Splicing Probe Instructions

Xtrace models GPU instruction scheduling and searches probe-specific choices for an insertion that preserves the kernel’s behavior while minimizing added issue time. It analyzes register liveness and hazards, then coordinates probes and trace collection across the kernel.

4.1 Modeling

  • Xtrace models in-order GPU issue: structural and fixed-latency data hazards use stalls, while variable-latency data hazards use scoreboards.The compiler encodes its hazard-resolving schedule in instruction control bits.Source: §4.1 · p. 5, Figure 7 · p. 6
  • A splicing jointly chooses dead-register assignments, a dependency-respecting instruction order, and control bits that resolve hazards.The search evaluates valid combinations and retains the plan with the lowest corrected overhead.Source: §4.1 · p. 6, §4.2 · p. 7

4.2 The Splicing Algorithm

  • Xtrace estimates overhead at the insertion address and corrects for dynamic data hazards using measured completion latencies.Its greedy alternative prioritizes eligible instructions with the highest dependency-chain price, omitting WAR and WAW dependencies in that ordering heuristic.Source: §4.2 · pp. 6–7
  • The optimal search enumerates probe instruction orders, register assignments, and permitted control bits, evaluating valid plans by corrected overhead.Dependencies and equivalent register assignments sharply reduce the search space; the Figure 8 probe has 22 instruction orders and 10 register partitions.Source: §4.2 · p. 7

4.3 Allocating the Probe’s Registers

  • Register-liveness analysis identifies dead periods for probe reuse and accounts for warp-specialized register caps and registers held by asynchronous operations.When dead registers are insufficient, Xtrace can raise the kernel’s register cap or downgrade to per-warp tracing.Source: §4.3 · pp. 7–8, Figure 9 · p. 8

4.4 Resolving the Probe’s Hazards

  • Xtrace resolves structural and data hazards with compiler hazard tables, encoding only remaining stalls and using free scoreboards where available.When scoreboards run short, it can wait early on an in-flight dependency to reclaim one, trading some instruction overlap for a free scoreboard.Source: §4.4 · p. 8

4.5 Tracing with Spliced Probes

  • INIT probes establish a thread’s trace-buffer slot and ring-buffer pointer; RECORD probes capture runtime values and write them to that slot.The runtime driver sizes the buffer to the launch, and each thread records into its own slot before the driver collects the trace.Source: §4.5 · pp. 8–9

5 Evaluation

Xtrace-guided optimization lets an agent match cuDNN on three FA-4 shapes and exceed it on a fourth, while improving FA-4 throughput by 5.2–13.3%. On FA-3, Xtrace helps the agent reach about 566 TFLOPS in fewer iterations than the alternative traces.

5 Evaluation

  • The evaluation asks whether higher-fidelity traces improve optimization, whether Xtrace preserves the kernel binary, and how much overhead its mechanisms add.Figure 10 depicts the FA-4 optimization guided by Xtrace traces of FA-4 and closed-source cuDNN SDPA.Source: §5 · p. 9, Figure 10 · p. 10

5.1 Experimental Setup

  • Xtrace supports 19 NVIDIA and AMD GPU architectures and mainstream GPU programming languages, and is publicly available as a pip package.The paper reports deployment at AI model development and serving companies.Source: §5.1 · p. 9
  • Xtrace evaluates trace-schedule, pipeline, and memory probes on H100, B300, and MI300X across LLM training and inference kernels, including closed-source cuDNN SDPA.Neutrino is the IR-level baseline; IKET is a binary-level replacement baseline extended to the workloads in the evaluation.Source: §5.1 · p. 9

5.2.1 Case 1: Optimizing FA-3 across Tracing Tools.

  • 14 iterations with Xtrace versus 54 with Neutrino and 71 with IKET brought the FA-3 agent to about 566 TFLOPS; the untraced agent needed 85.The Xtrace trace localized the producer’s wait to 1,298 ns of a 1,435 ns window; Figure 11 presents the optimization comparison.Source: §5.2.1 · p. 10, Figure 11 · p. 10

5.2.2 Case 2: Optimizing FA-4 against cuDNN Trace

  • 5.2–13.3% throughput gains let optimized FA-4 match cuDNN on three of four shapes and run 5.2% faster on the fourth.The agent’s changes address CTA residency, cross-group tile ordering, tile pairing, and an unnecessary final-tile wait.Source: Figure 10 · p. 10, §5.2.2 · p. 11

5.3 Binary Similarity

Xtrace preserves the compiled kernel’s instructions substantially better than Neutrino and IKET. The section also compares probe overhead across trace types and launch sizes, and examines dependency waits around probes.

5.3 Binary Similarity

  • Xtrace changes no basic blocks and preserves 94–98% of original instructions, while Neutrino rewrites 54–92% and IKET rewrites 5–87%.Xtrace’s 2–6% changed instructions are branch instructions; on FA-3 it retains all 85 basic blocks, compared with 21% preserved by Neutrino and 46% by IKET.Source: §5.3 · p. 11

5.4 Tracing Overhead

  • Under trace-pp, Xtrace adds 0.9–2.7% duration, versus 5.6–12.5% for Neutrino and 1.3–3.8% for IKET.Trace-pp records timestamps at every boundary of the kernel’s pipeline stages.Source: §5.4.2 · p. 12
  • On FlashMLA’s innermost-loop memory accesses, trace-mem adds 0.7% with Xtrace, versus 36.9% with Neutrino and 20.0% with IKET.Across kernels, trace-mem overhead is 0.7–5.4% for Xtrace, 2.0–75.6% for Neutrino, and 0.5–28.4% for IKET.Source: §5.4.2 · p. 12
  • As FA-3 and FA-4 launches grow from 32 to 2,048 CTAs and probe records increase 249×, Xtrace overhead stays at or below 3.5%.Neutrino reaches 13.9% and IKET 5.1%; vendor IKET crashes on the two largest FA-3 launches.Source: §5.4.2 · p. 12
  • Under trace-pp, probe dependency waits account for 2.3–11.3% of kernel waits with Xtrace, 2.4–19.3% with Neutrino, and 0.8–16.5% with IKET.In some cases Neutrino and IKET wait less because pre-compilation scheduling hides waits, but their binaries are substantially rewritten; Xtrace removes an unnecessary store-scoreboard WAR dependency and retains the unavoidable register RAW dependency.Source: §5.4.3 · pp. 12–13, Figure 15 · p. 13

5.5 Reusing Dead-Period Registers

Xtrace’s dead-register reuse supplies probe registers across the evaluated splices without increasing kernel register budgets. The section also reports the cost of analyzing and searching for splice plans.

5.5 Reusing Dead-Period Registers

  • Dead-period register reuse succeeds in all 53 splices across three NVIDIA kernels without fallback or added registers, including all 45 trace-mem splices.Reuse also succeeds at peak register pressure; no measured kernel required the fallback of raising its register budget.Source: §5.5 · p. 13

5.6 Splicing Time

  • At median, binary analysis takes 0.33–0.68 seconds per splice, while plan search takes 2.6–4.9 milliseconds, under 2% of splice time.Xtrace-g searches in 2.5–4.9 milliseconds and produces plans with the same kernel cost as the optimal plans.Source: §5.6 · p. 13, Figure 16 · p. 13

6 Conclusion

Xtrace splices probes directly into compiled GPU kernel binaries to minimize compile-time interference and runtime overhead. On production LLM kernels, it preserves most original instructions with low overhead and also traces closed-source vendor kernels.

6 Conclusion

  • Xtrace preserves 94–98% of original instructions and adds 0.9–2.8% overhead on production LLM kernels, while its traces guide a coding agent and extend to closed-source vendor kernels.The conclusion reports these results as the outcome of binary-level splicing.Source: §6 · p. 13

A.1 Trace Result of Closed-Sourced Vendor Kernels.

Across three closed-source libraries, Xtrace adds 2.1–8.3% to kernel time, deletes no instructions, and leaves basic-block counts unchanged. Its traces characterize TensorRT-LLM GEMM, cuDNN SDPA, and cuBLAS GEMM across varied launch shapes.

A.1 Trace Result of Closed-Sourced Vendor Kernels.

  • Across three closed-source libraries, Xtrace adds 2.1–8.3% to kernel time, deletes no instruction, and leaves basic-block counts unchanged.The 2–5% of instructions not preserved verbatim are branch and reconvergence targets moved by insertion.Source: §A.1 · p. 17
  • Across five cuDNN SDPA launch shapes with measured baselines, Xtrace adds 2.6–3.1% to kernel time.The sixth shape uses multiple tiles per CTA, so its figure is intended for interpreting turns rather than pipeline ends.Source: §A.1 · p. 17
  • For cuDNN SDPA, doubling the batch-2 launch from 64 to 128 CTAs nearly doubles the SoftMax prologue from 1989 to 3984 ns, while the MMA turn body changes from 606 to 641 ns.The median CTA grows from 11.4 to 14.0 µs, with the 2.6 µs difference almost entirely in the prologue.Source: §A.1 · p. 18
  • At batch 2, sequence length 2048, and 8 heads, cuDNN SDPA has 128 CTAs and a 23.5 µs median CTA; Xtrace costs 3.1% on this longest launch.MMA warps are in turn stretches 64% of the time, while SoftMax warps spend 73% outside marked stretches.Source: §A.1 · p. 18
  • Xtrace adds 8.3% to TensorRT-LLM GEMM kernel time and preserves 96% of its instructions while deleting none.The dequantize warps spend 63% of their time unpacking NVFP4 to FP8, pacing the K loop.Source: §A.1 · p. 20
  • In the TensorRT-LLM trace, dequantize warps spend 63% of their time in K turns, versus 27% for MMA and 10% each for the two TMA warps.This indicates that NVFP4-to-FP8 unpacking, rather than Tensor Core work or loads, paces the loop.Source: §A.1 · p. 20
  • Across five cuBLAS GEMM problem sizes, median CTA times are 9.3, 14.0, and 24.6 µs at M=N=2048 as K grows from 1024 to 4096, then 26.5 and 49.2 µs at larger M and N.The producer spends 49% of its time in K-loop turns on average; epilogue warps spend 79% in the first store.Source: §A.1 · p. 20
  • Across cuBLAS GEMM sizes, the first drain grows from 5.9 to 21.2 µs as K rises from 1024 to 4096, while the other seven drains stay near 200 ns at K=2048.With multiple tiles per CTA, the producer turns lengthen to 278–281 ns while MMA turns remain 76–86 ns.Source: §A.1 · pp. 21–22

A.2 Per-Vendor Splicing Details.

This appendix describes how Xtrace instantiates the execution model from §4.1 on each vendor’s GPUs.

A.2 Per-Vendor Splicing Details.

  • The appendix’s scope is vendor-specific instantiation of the §4.1 execution model.It does not state the vendor-specific mechanisms in this overview passage.Source: §A.2 · p. 23
  • The execution model being instantiated is the one introduced in §4.1.The passage identifies that section as the model’s source.Source: §A.2 · p. 23
  • The appendix organizes its details around GPU vendors.Its stated subject is how Xtrace applies the model on each vendor’s GPUs.Source: §A.2 · p. 23

A.2.1 NVIDIA GPUs.

NVIDIA GPUs issue instructions in order and rely on compiler-encoded control signals to resolve hazards. Xtrace also reduces timestamping cost by recording per-SM clocks and calibrating them to a global timeline.

A.2.1 NVIDIA GPUs.

  • On NVIDIA GPUs, hardware issues instructions in order and does not interlock the register file or pipes, so the compiler encodes hazard resolution in instruction control signals.Xtrace therefore instantiates the full execution model for NVIDIA GPUs.Source: §A.2.1 · p. 23
  • Xtrace uses the per-SM clock for RECORD probes because it reads in about 10 ns, has one-cycle granularity, and needs only one register for typical kernel durations.A 32-bit per-SM capture wraps only after 1.4 to 3 seconds, longer than most kernels run.Source: §A.2.1 · p. 23
  • Because per-SM clocks are unsynchronized, Xtrace reads global and per-SM clocks at each block’s start and end to map SM cycles onto a global nanosecond timeline.This avoids the global clock’s hundreds-of-nanoseconds read latency at every RECORD probe.Source: §A.2.1 · p. 23

A.2.2 AMD GPUs.

AMD GPUs resolve most hazards in hardware, so Xtrace uses a reduced execution model, applying compiler-style padding and vendor-specific handling of register classes, counters, and timers.

A.2.2 AMD GPUs.

  • AMD instructions issue in order, but most register dependencies are interlocked in hardware; Xtrace pads remaining hazards using compiler rules and leaves kernel wait states unchanged.CDNA uses s_nop for waits; RDNA3 and later may use s_delay_alu as an optional spacing hint.Source: §A.2.2 · p. 23
  • Xtrace tracks AMD VGPR, SGPR, and AGPR liveness, keeping accumulation registers live through matrix instructions and adapting probe budgets when dead registers are insufficient.On RDNA4, it places INIT after s_alloc_vgpr so the probe uses the valid runtime register budget.Source: §A.2.2 · p. 23
  • On AMD, spliced memory operations enter per-wave completion counters, but Xtrace leaves kernel s_waitcnt instructions unchanged because the added probe operations do not release them early.Xtrace accepts the resulting timing cost rather than rewriting wait thresholds.Source: §A.2.2 · p. 24
  • On CDNA3, timestamp reads require lgkmcnt(0), which also drains in-flight LDS and scalar memory operations, so Algorithm 1 reads the timer before waiting prior to the store.The timer read is slower than the register clock used on NVIDIA GPUs.Source: §A.2.2 · p. 24
Loading 2609.28769v1…