PaperlayerView exact revision on arXiv

Source-linked AI summary

Argus: Orchestrating Cross-Layer GPU Performance Measurements around Semantic Regions

Jianzhu Yao, Yue Guan, Srivatsan Ramesh, Yuanwei Fang, Jian Jiao, Boda Li, Yueming Hao, Xinwei Qiang, Pramod Viswanath, Yufei Ding, Bill Yoshimi, Alexey Loginov, Shane Nay, Adnan Aziz

arXiv:2609.12299v1cs.DCcs.PF

TL;DR

GPU performance evidence is fragmented across profilers that expose different execution objects, making region-level questions require manual measurement and attribution work. Argus plans interference-aware, cross-backend measurements around semantic regions and joins the resulting evidence. Across its evaluations, it improves kernel-optimization outcomes and supports cross-level overlap tuning.

  • Problem

    GPU developers and optimizers need region-level evidence, but profiler views use different execution objects and answering region questions requires coordinating measurements and attribution across them.

    Source: §1 · pp. 1–2

  • Method

    Argus accepts region, signal, and scope queries, preserves region identity across program variants, and orchestrates interference-aware measurements and cross-run evidence synthesis.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Argus improves 39/44 persistent-GEMM and attention configurations, raises AlphaEvolve’s geometric-mean speedup from 5.4% to 8.9%, and supports cross-level PGO throughput gains averaging 7% across five multi-GPU settings.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Argus provides a human- and machine-consumable substrate for region-level performance evidence, with reports that retain measurement provenance and attribution ambiguity.

    Source: §1 · p. 3, §6 · p. 12

    Count-weighted attribution estimates metric contributions rather than measuring them directly, and association entropy indicates ownership sharing rather than an error bound.

    Source: §3.5 · p. 6

Abstract

from arXiv · show

GPU developers and automated optimizers need performance evidence for semantic code regions--such as neural-network operator implementations and pipeline stages--but this evidence is fragmented across profiling tools. Answering a region-level question can require manually constructing probes and program variants, isolating interfering measurements, and mapping evidence to regions and execution contexts. We present Argus, a region-centric measurement planner and runtime that automates this workflow. Clients identify regions with boundary markers and select signals and execution scopes. Argus preserves region identity across compilation, execution, and measurement variants, constructs interference-aware multi-run plans, and orchestrates transformations and profiling across backends. It joins compiler-, hardware-, and system-level evidence using region identity and dynamic execution context, producing reports that record measurement origins and attribution ambiguity. We evaluate Argus across agentic kernel optimization, persistent megakernel optimization, and cross-level PGO. Across 44 persistent-GEMM and attention configurations, Argus improves 39/44 cases and raises AlphaEvolve's geometric-mean speedup from 5.4% to 8.9%. On a persistent TinyLlama-1.1B decode megakernel, an optimization agent reaches 1.65 ms/token with Argus versus 4.92 ms/token without it, producing a kernel $2.1\times$ faster than PyTorch with CUDA Graphs. Finally, Argus-guided cross-level PGO improves compute--communication overlap, increasing throughput by 7% on average across five multi-GPU settings.

1 Introduction

Argus makes semantic regions the unit for GPU performance questions that otherwise require reconciling fragmented measurements and execution contexts. It plans and runs interference-aware measurements across profiling backends, then joins the evidence into region reports.

1 Introduction

  • Region-level tuning must reconcile compiler timing, machine behavior, system timelines, and execution context; issue timing captures only enqueue overhead, while forced completion perturbs overlap.Shared helper instructions also make instruction addresses alone insufficient to identify the active region.Source: §1 · p. 1
  • For warp-specialized GEMM, Argus separates reference tracing, completion probes, dynamic mapping, and hardware-metric collection to report timing, attributed metrics, and exposed latency.The example covers a mainloop timeline and wait-region duration and synchronization activity over 16 iterations.Source: Figure 1 · p. 2
  • Argus lets clients request signals for a region and scope, then plans transformations, profiling runs, and cross-run joins across heterogeneous backends.Its reports preserve region identity and execution context while recording measurement origins and attribution ambiguity.Source: §1 · p. 2
  • Argus improves 39/44 persistent-GEMM and attention configurations, raising AlphaEvolve’s geometric-mean speedup from 5.4% to 8.9%.On persistent TinyLlama-1.1B decode, an agent reaches 1.65 ms/token with Argus versus 4.92 ms/token without it; cross-level PGO improves throughput by 7% on average across five multi-GPU settings.Source: §1 · p. 3

2 Background

GPU optimization units span abstraction levels and may not remain contiguous or uniquely identifiable in machine code. Profilers expose complementary but differently indexed evidence, while asynchronous measurements can require separate runs.

2 Background

  • Optimization targets may be fused operators, loop stages, warp-role paths, or handoffs, and lowering can distribute their code or share it across execution contexts.Inlining, unrolling, software pipelining, and code sharing can break source-contiguous correspondence.Source: §2 · p. 3
  • Compiler profilers expose IR structure, machine profilers report PC-keyed metrics, and system profilers capture launches and timelines; these views do not directly name the same semantic unit.Answering one performance question can therefore require coordinating backends and reconciling evidence.Source: §2 · p. 3
  • Asynchronous operations separate issue from completion, so some quantities require altered instrumentation or synchronization and cannot always be measured in the reference run.These constraints make cross-backend profiling a multi-run measurement problem.Source: §2 · p. 3

3 Argus

Argus turns region-and-scope queries into measurement plans and reports, preserving actionable region identity across lowering and connecting evidence gathered in distinct runs. Its planner closes signal dependencies, discovers operation-specific targets, and separates measurements whose effects conflict.

3.1 Overview: From a Region Query to a Region Report

  • Clients request region signals and scope without enumerating operations or profiler runs; Argus expands the query into analysis, transformations, collections, and synthesis.For the GEMM example, the request covers wait-region duration and synchronization activity plus a mainloop asynchronous timeline over 16 iterations.Source: §3.1 · p. 3

3.2 Region Semantics and Identity

  • A region is a named, boundary-marked optimization target, while scope selects its dynamic executions, such as a warp role or iteration window.Stable IDs and identifiable begin/end transitions link regions across lowering and measurement variants.Source: §3.2 · p. 4

3.3 Planning Region Measurements

  • Each requested signal resolves to a reusable recipe that specifies evidence, prerequisite analyses, transformations, scope coordinates, collection effects, and synthesis rules.The planner binds recipes to a program version, region, and scope, then recursively adds prerequisite tasks.Source: §3.3 · p. 4
  • Compiler discovery identifies asynchronous sites and their completion conditions; TMA probes recover the barrier phase so a wait targets the selected iteration.The planner inserts the completion wait after issue and joins only that interval to the matching reference issue, because forced waits can perturb later overlap.Source: §3.3 · p. 5
  • One query can expand to 80 operation–iteration probes, which Argus partitions when instrumentation or forced waits could invalidate other measurements.The client selects the region and scope once rather than enumerating operations, completion conditions, iterations, or profiler runs.Source: §3.3 · p. 5

GPU 1

Argus executes measurement plans as dependency-driven work while isolating incompatible GPU collections and retaining versioned inputs and outputs. It synthesizes qualified region reports that distinguish observed, attributed, and derived evidence and retain ambiguity information.

GPU 1

  • The runtime dispatches input-ready tasks while serializing incompatible collections on each device, and versioned artifacts retain task and input provenance for synthesis, reuse, and reruns.GPU collections run in isolated workers; independent transformations, parsing, and synthesis can execute concurrently.Source: Figure 3 · p. 5, §3.4 · p. 6

3.4 Runtime: Enforcing Multi-Run Plans

  • Argus represents plan nodes as transform, collect, synthesize, or control tasks, and expands dependent branches when compiler discovery reveals new targets.Task bindings preserve program version, region, scope, backend configuration, and dependencies.Source: §3.4 · pp. 5–6

3.5 Synthesizing Qualified Region Reports

  • Reports qualify fields as observed, attributed, or derived and join evidence only when program versions and scope coordinates are compatible.For a probe, the matching reference is the same operation in the same iteration, not timestamps compared across executions.Source: §3.5 · p. 6
  • Count weighting estimates contributions from shared PCs, while association entropy indicates ownership sharing rather than an error bound.For the GEMM query, wait-region duration is observed, machine-level evidence is attributed, and mainloop exposed-latency distributions are derived.Source: §3.5 · p. 6

4 Evaluation

Argus is evaluated as an interface for agentic kernel optimization, as a diagnosis tool for persistent megakernels, and as a substrate for cross-level PGO; the evaluation also measures system costs. Across these settings, region-scoped evidence improves optimization outcomes and helps guide changes at both operator and scheduler levels.

4.1 Agentic Kernel Optimizers as Clients

  • Argus improved 39/44 persistent-GEMM and attention configurations, raising AlphaEvolve’s geometric-mean speedup from 5.4% to 8.9% over the original kernels.Configurations reaching at least 10% speedup increased from 2/44 to 19/44.Source: Figure 4 · p. 7, §4.1 · p. 7
  • All five reproduced agentic workflows benefited from Argus under the same 25-trial budget, while retaining their existing search logic.For example, AlphaEvolve improved from 8% to 21.9%, and CUDAForge changed from a −2.7% regression to a 3.5% gain.Source: Figure 6 · p. 7, §4.1 · p. 8
  • A region report guided asynchronous scheduling changes in persistent GEMM; one configuration improved performance by 13.3% through async-operation reordering based on profiling traces.The case study’s strategy prioritized the longest asynchronous operations and signaled barriers immediately after async MMA issue.Source: Figure 7 · p. 8, §4.1 · p. 8

4.2 Scaling Diagnosis to a Persistent megakernel

  • The Argus-guided TinyLlama-1.1B decode agent reached 1.65 ms/token versus 4.92 ms/token without Argus, and its endpoint was 2.1× faster than CUDA-Graph PyTorch.Region-attributed evidence guided selective de-inlining, redundant-barrier removal, and direct phase chaining in the persistent megakernel.Source: §4.2 · pp. 9–10, Figure 9 · p. 9, Figure 8 · p. 9

4.3 Cross-level overlap PGO

  • Across five GEMM cases, cross-level PGO improved throughput over PyTorch baselines by an average of 7%.The overlap model and end-to-end profiling identified the same high-performance band; the exact optimum shifted with problem shape and network cost.Source: §4.3 · p. 10, Figure 12 · p. 10, Figure 13 · p. 11

4.4 System Cost

  • For fixed region-occurrence counts, reference-tracing overhead fell as useful work increased; batching eight probes reduced turnaround to 1.00 s/probe, a 2.97× improvement.Workspace grew linearly at 2.08 MiB/probe.Source: §4.4 · p. 11, Figure 14 · p. 11

5 Related Work

Argus distinguishes itself from GPU profiling and optimization systems by using modifiable code regions as a shared attribution space. It plans and coordinates measurements across heterogeneous tools without prescribing how optimizers use the resulting evidence.

GPU Profiling and Attribution

  • Argus uses developer-modifiable code regions to connect compiler, hardware, instrumentation, and system-profiler evidence while preserving region identity across lowering and execution.Prior systems provide complementary views, including compiler-centric timing, instruction-level metrics, programmable probes, calling contexts, and framework-level correlations.Source: §5 · p. 11

Programmable Observability and Measurement Orchestration

  • Argus builds dependency- and interference-aware multi-run plans from region-level requests and preserves measurement lineage through runtime orchestration and synthesis.Prior work includes programmable instrumentation, declarative tracing, event correlation, controlled perturbations, and contextual measurement interfaces.Source: §5 · p. 11

Agentic GPU Kernel Optimization

  • Argus complements agentic kernel optimizers with reusable region-attributed evidence for human developers, compiler PGO passes, and automated optimizers.Unlike optimization-policy and kernel-generation work, Argus does not prescribe how evidence is used to search the optimization space.Source: §5 · p. 11

6 Conclusion

Argus addresses fragmented GPU performance evidence by making semantic regions a common unit across compilation, execution, and measurement. Its evaluations show improvements for agentic optimization, persistent megakernel optimization, and cross-level PGO.

6 Conclusion

  • Argus raises an existing optimizer’s geometric-mean speedup from 5.4% to 8.9% across 44 kernels, guides a 3.8× speedup on a persistent LLM decode megakernel, and supports 7% average throughput improvement across five multi-GPU settings.The throughput improvement is over PyTorch baselines and comes from compute-communication overlap tuning.Source: §6 · p. 12
  • Argus makes semantic regions a common attribution space across compilation, execution, and measurement, coordinating heterogeneous profilers into interference-aware multi-run plans and unified region reports.The reports preserve measurement origins and expose attribution ambiguity.Source: §6 · p. 12
  • The conclusion characterizes GPU performance debugging as fragmented because developers optimize semantic regions while profilers expose different entities requiring explicit reconciliation.Source: §6 · p. 12

A Overlap PGO Tool Evaluation Setup

The PGO setup identifies four GEMM shapes for evaluation and lists the shapes used in its tests.

A Overlap PGO Tool Evaluation Setup

B Trace Example of the Asynchronous Timeline Profiling Signals

The asynchronous timeline example combines reference traces with aggregated operation latencies and percentile distributions across iterations. It organizes traces by warp group to support comparison of execution timing and variability.

B Trace Example of the Asynchronous Timeline Profiling Signals

  • Sixteen lanes separate producer and two consumer warp groups, with reference traces anchoring alignment of later probe measurements to unmodified execution.The example shows the first six producer iterations and first five consumer iterations; TMA loads and WGMMA operations occupy separate lanes.Source: B Trace Example of the Asynchronous… · p. 13, Figure 15 · p. 14
  • Mean lanes show average exposed operation times, while percentile bands from minimum through 80th percentile display latency variability across sampling runs.The percentile colors are dark blue, light blue, yellow, orange, and red, respectively.Source: B Trace Example of the Asynchronous… · p. 13
  • The trace visualization combines fine-grained measurement, statistical aggregation, and visual correlation to inspect asynchronous execution, validate overlap patterns, and identify opportunities traditional profilers miss.The example is a warp-specialized persistent GEMM with one producer warp and two consumer warps.Source: B Trace Example of the Asynchronous… · p. 13, Figure 15 · p. 14
Loading 2609.12299v1…