Source-linked AI summary
VortexSynth: Profile-guided benchmark reconstruction for GPU architecture research
Huanzhi Pu, Chulhyung Park, Chihyo Ahn, Ramyad Hadidi, Hyesoon Kim
TL;DR
Vendor-tailored workloads and library kernels can be difficult to run on research architectures, making representative benchmarks costly to migrate. VortexSynth reconstructs and iteratively refines target kernels using output checks and profile feedback. Its 26-pair Vortex robotics suite passes both equivalence checks, and case studies report reduced profile distance.
Problem
Vendor-tailored workloads and library kernels may not run on research architectures, and adapting kernels or replacing unavailable library implementations requires manual effort.
Source: §1 · p. 1Method
VortexSynth profiles reference workloads, synthesizes target-language kernels, checks output equivalence, and iteratively refines candidates using profile differences.
Source: §1 · p. 1, §6 · p. 12Results
The 26 VX-Robotics Vortex/CUDA pairs all pass both equivalence checks, and refinement lowers profile distance in the detailed case studies.
Source: §6 · p. 12Takeaways & Limitations
VX-Robotics provides reusable kernels for Vortex architecture research, with the framework intended to support benchmark construction on Vortex and other open GPUs.
Source: §6 · p. 12The CUDA proxy can carry Vortex-imposed implementation choices, so measured distance may mix workload mismatch with existing architectural constraints.
Source: §4.5 · p. 10
Abstract
from arXiv · showhide
GPU architecture research requires representative benchmarks that can execute on new designs. However, many vendor-tailored workloads and library kernels cannot run directly on research architectures because their implementations are closed-source or rely on programming models unsupported by the target architecture. Migrating existing kernels requires manual effort to adapt their implementations and validate their behavior. This creates a need to automate workload migration to target platforms. We present VortexSynth, a framework that uses profile-guided benchmark reconstruction to construct GPU benchmarks from reference workloads. Our workflow selects hot kernels through profiling, synthesizes implementations in the target programming language, verifies output equivalence, and measures profile differences using a translated proxy implementation. The resulting profile signature gap is supplied to an agent for iterative refinement. We evaluate the method on Vortex, an open-ISA GPGPU, using robotics as the application domain. All active kernel pairs pass both output-equivalence checks, and the case studies show that iterative refinement can reduce profile distance.
1 Introduction
GPU architecture research needs representative benchmarks as workloads and target designs change, but vendor-tailored workloads and library kernels may require substantial manual adaptation or replacement. VortexSynth reconstructs profiled hot kernels for target platforms and refines them using profile feedback.
1 Introduction
- Open-source GPU designs such as Vortex support reproducible research and transparency in architectural exploration.Vortex is an open-source GPGPU based on RISC-V.Source: §1 · p. 1
- Benchmark preparation recurs as applications and architectures change, because researchers need representative workloads that run on their specific designs.New models, operators, numerical formats, and target architectures can change which kernels benchmarks must represent.Source: §1 · p. 1
- Closed-source library kernels can require replacement implementations, so porting surrounding code alone is insufficient for making those computations available as open-GPU benchmarks.Hardware, compiler, and runtime changes can then be evaluated using representative application workloads.Source: §1 · p. 1
- VortexSynth selects hot kernels, synthesizes target-language implementations, checks output equivalence, and uses profile differences to guide iterative refinement.The focus on kernels keeps benchmarks suitable for simulation, where full-model inference can take days.Source: §1 · pp. 1–2
- The paper contributes the VX-Robotics suite of 26 Vortex kernels from robotics workloads, with reusable Vortex/CUDA pairs, reference profiles, and output-validation artifacts.VortexSynth synthesizes Vortex kernels without access to the original source code.Source: §1 · p. 2
2 Background
The background introduces Vortex and its benchmark model, describes the robotics workloads used as references, and motivates iterative kernel migration with correctness and profile feedback.
2.1 Vortex GPGPU
- Vortex is a full-stack open-source RISC-V GPGPU research platform with SIMT execution, OpenCL support, simulator and RTL flows, and a modifiable software stack.Applying the method to another GPU requires that platform’s programming interface, implementation constraints, and validation tools.Source: §2.1 · p. 2
2.2 Vortex Benchmark
- A Vortex benchmark targets the Vortex software stack, compiling C/C++ device kernels into RISC-V GPGPU instructions rather than NVIDIA code or library calls.The listed intrinsics support thread launch, lane masking, barriers, local memory, and tensor-core multiply-add.Source: §2.2 · p. 2
2.3 Robotics Workloads
- The reference applications are ACT, Diffusion Policy, and π0, spanning image-and-transformer policies, iterative action denoising, and vision-language-action generation.Their computations include attention, dense matrix operations, or repeated temporal convolutions.Source: §2.3 · p. 2
2.4 LLM
- Prior work motivates iterative migration because generated kernels may be incorrect after one translation; VortexSynth additionally supplies measured profile differences to guide revisions.The cited work describes compiler diagnostics and execution results as feedback for correcting translation errors.Source: §2.4 · p. 2
3 VortexSynth:Profile-guided benchmark reconstruction
VortexSynth profiles reference workloads, reconstructs hot kernels as paired CUDA and Vortex candidates, checks their outputs, and iteratively retains revisions that reduce profile-signature distance.
3 VortexSynth:Profile-guided benchmark reconstruction
- Each refinement pass generates CUDA and Vortex candidates; the framework keeps a pair only when profile distance falls and locks the retained candidates when optimization axes are exhausted.The locked Vortex implementation becomes the reconstructed benchmark.Source: §3 · p. 3
3.1 Reference Program Context
- The workflow profiles operator calls and dispatched kernels, extracting tensor shapes, data types, engine class, block size, and launch grid as reference context.Kernel-name properties are used only when measured launch geometry agrees; undocumented naming conventions are left for search.Source: §3.1 · p. 3
3.2 Source Profile Signature
- The profile signature records hot-kernel rank, cumulative time, launch parameters, and metrics grouped by hardware pressure; its distance averages normalized metric differences and omits unmeasurable metrics.The signature captures resource-use patterns, not exact instruction schedules, stall reasons, bank conflicts, or register allocations.Source: §3.2 · pp. 3–4
3.3 Refinement Sandbox
- The sandbox fixes allowed edits, constructs, and correctness requirements, while its twelve portable optimization axes cover data movement, reuse, execution engine, result commitment, and utilization.Some NVIDIA-oriented optimizations are represented with Vortex mechanisms, such as software prefetching for asynchronous copies and FP16 for TF32 tensor-core paths.Source: §3.3 · pp. 4–5
3.4 Candidate Generation
- Each hot kernel has its own revision chain: CUDA candidates are refined under fixed reference properties, and corresponding Vortex candidates use aligned constructs for each optimization axis.The CUDA candidate is measured on an NVIDIA GPU as a proxy for the paired implementation strategy.Source: §3.4 · p. 5
3.5 Output Equivalence
- Every revision must pass source-output and CUDA-to-Vortex candidate equivalence checks before its profile distance is measured.Tolerances depend on precision path and accumulation differences; tensor-core paths allow wider relative tolerances, including for the TF32-to-FP16 change.Source: §3.5 · p. 5
3.6 Profile-Guided Refinement
- Refinement starts from the best retained design, usually changes one axis at a time, and rejects failed checks or revisions that do not lower distance beyond the 0.005 noise margin.The agent receives per-metric distances for subsequent revisions and may combine axes after single-axis attempts stop improving the profile.Source: §3.6 · p. 6
4 Evaluation
The evaluation reconstructs robotics kernels as Vortex benchmarks and checks their output agreement and profile behavior. CUDA-first refinement reduces profile distance, while reversing the refinement order produces larger distances for most metrics.
4 Evaluation
- The evaluation tests ACT, Diffusion Policy, and Pi0 at batch size one, then checks coverage, output equivalence, optimization-axis responses, and refinement order.It also follows a Pi0 kernel through refinement and reverses the refinement order to test whether reference-GPU refinement makes the profile signal more usable.Source: §4 · p. 6
4.1 Kernel Coverage
- The 26-kernel suite covers linear layers, matrix multiplication, and 1-D and 2-D convolution, using the top five GPU-time kernels from each workload on each platform.Table 6 lists each benchmark’s reference kernel, GPU-time share, precision path, revision count, and final signature distance.Source: §4.1 · p. 7, Table 6 · p. 8
4.2 Output Equivalence Results
- All 26 benchmarks pass both output checks, with the largest reference-operation errors of 1.6×10−6 absolute for FP32, 1.1×10−2 relative for TF32, and 4.9×10−4 absolute for BF16.Across CUDA and Vortex, all nine FP32 pairs pass; TF32-to-FP16 pairs have the largest relative error, 9.5×10−2, and BF16 pairs reach 3.2×10−2 relative error on near-zero outputs while passing the combined tolerance.Source: §4.2 · p. 7
4.3 How Optimization Axes Move the Signature
- Optimization axes can affect pressure groups beyond their intended target, and the useful direction depends on the reference profile.Staging and lane mapping have larger mean responses in traffic metrics than in work per byte; read-once staging can move L1 hit rate in opposite directions across workloads. Register tiling raises arithmetic intensity but lowers occupancy, while software prefetching raises DRAM bandwidth yet all seven 1-D-convolution applications are reverted.Source: §4.3 · pp. 7–8
4.4 Case Study: Pi0 Attention Matmul
- Reversing the second operand’s lane mapping makes consecutive lanes access separate memory sectors, reducing coalescing to 29% versus the reference’s 37%, lowering signature distance to D=0.0820, and preserving output equivalence.The final kernel’s arithmetic intensity stays near a quarter of the reference value, and its L2 hit rate reaches 76% against 91%.Source: §4.4 · p. 9
4.5 CUDA-First versus Vortex-First Refinement
- Vortex-first refinement has higher distance for most metrics; DRAM bandwidth rises from 0.115 to 0.463 and coalescing from 0.064 to 0.349, supporting CUDA-first refinement.The authors attribute the less-clear feedback to mixing behavioral mismatch with effects of Vortex-specific adaptations.Source: §4.5 · pp. 10–11
5 Related Work
Prior retargeting and cloning approaches do not meet the paper’s combined aim: reconstructing a specific observed GPU kernel’s behavior on another ISA while preserving its outputs. VX-Robotics and VortexSynth address this gap with profiled robotics kernels and profile-guided LLM refinement.
5.1 Compiler-Based Retargeting and Workload Cloning
- Source-level retargeting preserves semantics when source and comparable primitives are available, whereas this work targets vendor binaries whose observed behavior must be reproduced.The passage notes that operator-level autotuners emit the fastest target kernel, not necessarily one resembling the reference kernel.Source: §5.1 · p. 11
- Prior workload clones reproduce measured application characteristics or GPU execution behavior, whereas this work also preserves outputs for substitution into the original stack.The target is a single GPU kernel specified by its profile signature and operator, on a different ISA.Source: §5.1 · p. 11
5.2 Robotics and ML Benchmark Suites
- Table 8 compares robotics, ML, and heterogeneous-computing suite scope; VX-Robotics reconstructs kernels from profiled robotics workloads as standalone Vortex benchmarks.The reconstruction uses output checks and profile-guided refinement.Source: §5.2 · p. 11, Table 8 · p. 11
5.3 Robotics Accelerators and Kernel Baselines
- Prior robotics accelerator studies cover VIO/ORB, sparse 3D, motion planning and dynamics, and edge learned inference, but usually focus on one robot type or algorithm family.The suite overlaps with kernels across these areas.Source: §5.3 · p. 11
5.4 LLM Kernel Generation
- KernelBench [37], TritonBench [27], and TritonGym assess generated-kernel quality; VortexSynth instead refines benchmark candidates derived from profiled workloads.KForge [39] leaves synthesis for closed-source kernels to future work, while Kevin and KernelCraft [5] [34] optimize speed rather than reference behavior; VortexSynth uses correctness checks and profile feedback without original source or extra model training.Source: §5.4 · pp. 11–12
6 Conclusion
VortexSynth reconstructs portable benchmarks from CUDA workloads without original kernel source, using output-equivalence checks and profile feedback to refine target kernels. The conclusion reports successful validation across 26 pairs and reduced profile distance in detailed case studies.
6 Conclusion
- VortexSynth uses workload context, output-equivalence checks, and profile feedback to generate and refine kernels without the original CUDA kernel source.The method is presented as reconstructing portable benchmarks from CUDA workloads.Source: §6 · p. 12
- All 26 Vortex/CUDA kernel pairs pass both equivalence checks, and detailed case studies show refinement lowers profile distance.The pairs come from ACT, Diffusion Policy, and Pi0 workloads profiled on H200 and Jetson AGX Orin.Source: §6 · p. 12
- VX-Robotics provides reusable kernels for Vortex architecture research, reconstructed from robotics workloads as Vortex/CUDA pairs.The paper says VortexSynth and the suite will be released as open source.Source: §6 · p. 12