PaperlayerView exact revision on arXiv

Source-linked AI summary

CompilerGym: Robust, Performant Compiler Optimization Environments for AI Research

Chris Cummins, Bram Wasti, Jiadong Guo, Brandon Cui, Jason Ansel, Sahir Gomez, Somya Jain, Jia Liu, Olivier Teytaud, Benoit Steiner, Yuandong Tian, Hugh Leather

arXiv:2109.08267v2cs.PLcs.AIcs.LGcs.PF

TL;DR

Compiler-optimization research faces a substantial engineering barrier and lacks the reusable infrastructure needed for fast experimentation. CompilerGym provides Gym-based environments and tools for compiler tasks; across its included environments, it offers larger search spaces and datasets and is 27× faster than prior works.

  • Problem

    Compiler research is hindered by complex experimental infrastructure, despite growing work on autotuning and learning-based compiler optimization.

    Source: §I · p. 1

  • Method

    CompilerGym exposes compiler-optimization tasks as Python Gym environments and provides tools, datasets, and infrastructure for experiments.

    Source: §I · p. 2

  • Results

    Across three compiler environments, CompilerGym offers larger search spaces and millions of training programs, and is 27× faster than prior works.

    Source: §I · p. 2

  • Takeaways & Limitations

    CompilerGym aims to lower the engineering barrier so researchers can experiment with production compiler problems regardless of compiler expertise.

    Source: §I · p. 1, §IX · p. 10

Abstract

from arXiv · show

Interest in applying Artificial Intelligence (AI) techniques to compiler optimizations is increasing rapidly, but compiler research has a high entry barrier. Unlike in other domains, compiler and AI researchers do not have access to the datasets and frameworks that enable fast iteration and development of ideas, and getting started requires a significant engineering investment. What is needed is an easy, reusable experimental infrastructure for real world compiler optimization tasks that can serve as a common benchmark for comparing techniques, and as a platform to accelerate progress in the field. We introduce CompilerGym, a set of environments for real world compiler optimization tasks, and a toolkit for exposing new optimization tasks to compiler researchers. CompilerGym enables anyone to experiment on production compiler optimization problems through an easy-to-use package, regardless of their experience with compilers. We build upon the popular OpenAI Gym interface enabling researchers to interact with compilers using Python and a familiar API. We describe the CompilerGym architecture and implementation, characterize the optimization spaces and computational efficiencies of three included compiler environments, and provide extensive empirical evaluations. Compared to prior works, CompilerGym offers larger datasets and optimization spaces, is 27x more computationally efficient, is fault-tolerant, and capable of detecting reproducibility bugs in the underlying compilers. In making it easy for anyone to experiment with compilers - irrespective of their background - we aim to accelerate progress in the AI and compiler research domains.

I. INTRODUCTION

CompilerGym addresses the engineering barrier to compiler-optimization research with reusable Gym-based environments and tools for experiments. Its architecture separates a Python frontend from compiler backends, and its included environments target three compiler-optimization problems.

I. INTRODUCTION

  • Compiler research is hindered by complex experimental infrastructure, while CompilerGym aims to let AI researchers work without compiler expertise and compiler experts integrate ML without AI expertise.The introduction situates this barrier alongside autotuning, machine learning, and reinforcement learning approaches to compiler optimization.Source: §I · p. 1
  • The contribution includes a Python library exposing compiler-optimization tasks as easy-to-use environments through the Gym API.The release also includes an integration toolkit, public leaderboards, web and command-line tools, and offline datasets comprising millions of performance results.Source: §I · p. 2
  • CompilerGym’s three environments target LLVM phase ordering, GCC flag selection, and CUDA loop-nest generation, with larger search spaces and 27× faster operation than prior works.They include millions of programs for training and support optimization for code size and runtime.Source: §I · p. 2
  • A standard interface enables integration with third-party libraries and reduces the engineering effort needed to create compiler experiments.The paper presents evaluation across autotuning and reinforcement-learning techniques as a demonstration of the platform’s research utility.Source: §I · p. 2

II. SYSTEM ARCHITECTURE

  • CompilerGym separates a Python frontend implementing Gym APIs and user-facing tools from compiler-specific backend integrations.The backend uses a client-server design, with RPC communication between frontend and compiler services.Source: §II · p. 2, Backend · p. 2

III. FRONTEND API AND TOOLS

CompilerGym formulates compiler-optimization tasks as Markov Decision Processes exposed through the OpenAI Gym interface. The interface supports different action, observation, and reward spaces, along with operators for stepping through and resetting an environment.

A. OpenAI Gym Environments

  • CompilerGym represents compiler-optimization tasks as Markov Decision Processes and exposes them as OpenAI Gym environments.The Gym interface provides five ingredients: action space, observation space, reward space, step, and reset.Source: A. OpenAI Gym Environments · p. 2
  • Action spaces can include discrete or continuous choices and may change between states, while observations can be compiler-analysis features, control-flow graphs, or IR strings.Environments can expose multiple observation spaces for selection or composition.Source: A. OpenAI Gym Environments · pp. 2–3
  • A step applies an action and returns a new observation, reward, and terminal-state signal; reset returns an initial observation.The user seeks an action sequence that maximizes cumulative reward, using approaches such as search or learned policies.Source: A. OpenAI Gym Environments · p. 3

B. API Extensions for Compiler Optimization

CompilerGym supplements Gym with compiler-specific APIs for benchmark datasets, environment state, and validation. These extensions support large benchmark collections and reproducibility checks, including detection of an LLVM nondeterminism bug.

1) Benchmark Datasets:

  • The dataset API manages collections of compiler benchmarks at million-scale and supports downloading, generating, compiling, iterating over, and running benchmarks.A benchmark is a program to optimize; datasets are collections of benchmarks.Source: 1) Benchmark Datasets · p. 3

2) State Serialization:

3) Validating States:

  • State validation detected a nondeterminism bug in an LLVM optimization pass, which was then removed from CompilerGym.Source: 3) Validating States · p. 3

4) Validating Semantics:

CompilerGym adds tools for validating results and efficiently exploring compiler environments. Its wrappers, command-line tools, and environment operations support customization and experimentation.

4) Validating Semantics:

  • Differential testing checks runnable benchmarks for correctness errors, and LLVM environments also use sanitizers to detect program logic errors.The integrated sanitizers are address, thread, and undefined-behavior sanitizers.Source: 4) Validating Semantics · p. 3

5) Lazy and batched operations:

  • Batched steps can run multiple actions together, while requested observation and reward spaces enable lazy computation and flexible space selection.These options can reduce backend work when observations or rewards are not needed at every step.Source: 5) Lazy and batched operations · p. 3

6) Lightweight deep copy operator:

  • The fork() operator creates independent deep copies for evaluating alternative actions, supporting search methods when compiler actions have no undo.The passage gives greedy search as an example: fork once per action, try each action, then select the greatest reward.Source: 6) Lightweight deep copy operator · p. 3

C. Customizing Environment Behavior

  • Composable wrappers configure action subsets, benchmark iteration, and derived observations; the LLVM -gvn-sink pass can also produce inconsistent output by sorting blocks by address.The wrappers support derived observations such as custom compiler analyses on compiler IR.Source: C. Customizing Environment Behavior · pp. 3–4

D. Command Line Tools

  • Command-line tools support parallelized searches, replay and validation of past runs, and code-free compiler interaction through an interactive shell.The shell includes inline documentation and tab completion.Source: D. Command Line Tools · p. 4

E. Web Service and CompilerGym Explorer

CompilerGym provides a web explorer, an offline state-transition dataset, and a shared runtime for compiler integrations. The runtime connects each compiler session to the Gym API while isolating compiler services from Python.

E. Web Service and CompilerGym Explorer

  • CompilerGym Explorer displays the search tree and updates it through a REST API as users explore; it also visualizes historical reward and observation trends.The trend views are intended to help identify interesting actions, support feature engineering and agent debugging, and serve as an educational tool.Source: E. Web Service and CompilerGym Explorer · p. 4

F. State Transition Dataset

  • The released state-transition database contains over 1M unique LLVM states and supports offline supervised and unsupervised learning tasks.The database is over 50 GB; its schema records action sequences, state observations, and transitions with rewards.Source: F. State Transition Dataset · p. 4, Figure 4 · p. 4

A. The CompilationSession Interface

IV. BACKEND RUNTIME AND INTERFACE

B. Compiler Service Runtime

  • The compiler service runs in a separate process and communicates with Python over RPCs, supporting recovery from crashes, remote architectures, and concurrent users without contention on Python’s global interpreter lock.The runtime also uses timeouts, graceful error handling, or retry loops for service operations.Source: B. Compiler Service Runtime · p. 5

V. ENVIRONMENTS

CompilerGym includes environments for compiler optimization tasks, including LLVM phase ordering. The LLVM environment exposes pass sequences, multiple optimization targets and observations, and over a million benchmarks excluding random program generators.

A. LLVM Phase Ordering

  • LLVM phase ordering is a challenging, high-dimensional optimization problem in which significant gains can be achieved.The choice and ordering of optimization passes affect the final binary’s quality.Source: A. LLVM Phase Ordering · p. 5
  • The LLVM action space contains 124 optimization passes, and an episode has no maximal length because passes may be repeated or omitted.Users estimate when further actions are unlikely to yield gains.Source: A. LLVM Phase Ordering · p. 5
  • LLVM rewards report changes in code size, binary size, or runtime, optionally scaled against gains from -Oz for size or -O3 for runtime.Code size is platform-independent and deterministic; binary size is platform-dependent and deterministic; runtime is platform-specific and nondeterministic.Source: Rewards · p. 5
  • The environment offers five LLVM observation spaces, from numeric counter features to sequential language models and graph-based program representations.It provides millions of programs across C, C++, OpenCL, and Fortran; excluding random generators, the benchmark total is 1,145,499.Source: A. LLVM Phase Ordering · p. 5, Datasets · p. 5, Table I · p. 6

B. GCC Flag Tuning

The GCC environment exposes command-line flag optimization through CompilerGym, with a finite configuration space and alternative action representations. It provides deterministic size rewards and several compiler-output observations.

B. GCC Flag Tuning

  • GCC 11.2.0 exposes 502 optimization options, yielding an optimization space of approximately 104461 configurations.The options include six -O<n> flags, 242 flags, and 260 parameterized flags; earlier GCC versions may expose smaller spaces because their parameter spaces are reported less clearly.Source: B. GCC Flag Tuning · p. 6
  • For GCC 11.2.0, the flat categorical action representation provides 2281 actions that modify option choices.A second representation directly exposes the optimization space as a list of integers, one per option.Source: Actions · p. 6
  • The GCC environment offers deterministic rewards for assembly size or object-code size, plus observations of instruction count, RTL, assembly text, or object code.The compiler can be selected by version, local path, or Docker image; the environment supports GCC versions 5 through 11.2 at the time of writing.Source: Actions · p. 6, Observations · p. 6, B. GCC Flag Tuning · p. 6

C. CUDA Loop Nest Code Generation

CompilerGym exposes CUDA loop-nest tuning through a cursor-based action space and FLOPs-based rewards. Its implementation includes packaged releases, documentation, and broad testing.

C. CUDA Loop Nest Code Generation

  • Point-wise CUDA tuning spans loop ordering, nesting, reuse, parallelization, and optional loop splitting in loop_tool’s representation.The representation decomposes BLAS-like routines into a DAG of arithmetic primitives before lowering them to an annotated loop tree.Source: C. CUDA Loop Nest Code Generation · p. 6
  • A cursor-based discrete action space lets users navigate loop hierarchies, resize loops, and assign loops to CUDA threads.Resizing an inner loop adjusts its parent and may induce automatically handled tail logic.Source: C. CUDA Loop Nest Code Generation · pp. 6–7
  • Rewards measure the loop nest’s benchmarked FLOPs per second, which varies by platform and is noisy.Observations expose cursor position and mode, plus a textual dump of the loop tree.Source: C. CUDA Loop Nest Code Generation · p. 7

VI. IMPLEMENTATION

  • CompilerGym’s implementation includes Python and C++ integrations, precompiled Linux and macOS releases, API documentation, and continuous testing.The unittest suite reports 85.8% branch coverage; daily fuzz and stress tests run through continuous integration.Source: §VI · p. 7, Binary Releases · p. 7, Documentation · p. 7

VII. EVALUATION

CompilerGym’s evaluation measures environment throughput and observation and reward costs. It reports a substantial throughput advantage over prior work and wide variation in operation times.

A. Computational Efficiency

  • Across cBench’s 23 programs, median step times for the fastest program, crc32, and slowest, ghostbench, differ by 560.3×.The cumulative density plot shows the distribution of step times varies widely within one dataset.Source: Figure 6 · p. 7, A. Computational Efficiency · p. 7
  • After an initial startup penalty, CompilerGym is 27× faster than an equivalent prior work while computing the same actions, observations, and rewards.Its server incrementally applies each optimization pass, while Autophase and OpenTuner re-read and parse the IR and apply the full pass sequence at each step.Source: A. Computational Efficiency · p. 7

B. Computational Efficiency of Observation Spaces

  • Observation-space times span 192× and reward-space times 4727×, exposing a tradeoff between observation cost and fidelity.The reward-time range motivates fast approximate proxy rewards and cost models.Source: B. Computational Efficiency of… · p. 7

C. Autotuning LLVM Phase Ordering

CompilerGym supports LLVM phase-ordering and GCC flag autotuning through a standard interface. The experiments describe search methods and show that LLVM autotuning can outperform default compiler heuristics when given a generous budget.

C. Autotuning LLVM Phase Ordering

  • LLVM experiments compare greedy search, LaMCTS, Nevergrad, OpenTuner, and random search on cBench for code size, binary size, and runtime.Each technique runs single-threaded for one hour per benchmark; runtime rewards use medians of three measurements during search and 30 for final values, with ten repetitions.Source: C. Autotuning LLVM Phase Ordering · pp. 7–8
  • CompilerGym’s standard interface makes third-party autotuning integration straightforward, and Table IV reports integration code and performance for three LLVM targets.The table compares size reduction against -Oz and runtime against -O3.Source: C. Autotuning LLVM Phase Ordering · p. 8, Table IV · p. 8
  • With a generous search budget, LLVM autotuning outperforms default compiler heuristics by tailoring configurations to benchmarks, and optimal configurations differ across benchmarks and targets.The LLVM experiments use a one-hour search budget and compare against -Oz for size reduction and -O3 for runtime.Source: C. Autotuning LLVM Phase Ordering · p. 8, Table IV · p. 8

D. Autotuning GCC Command Line Flags

  • GCC 11.2.0 exposes 502 optimization settings; random search, hill climbing, and a genetic algorithm are compared on CHstone with 1,000 compilations per search.Table V reports geometric-mean object-code size across benchmarks, averaged over three searches, against -Os.Source: D. Autotuning GCC Command Line Flags · p. 8, Table V · p. 9

E. Autotuning CUDA Loop Nests

CompilerGym’s CUDA environment explores loop-nest configurations, while a graph neural network learns an instruction-count cost model from transition data. Reinforcement-learning agents generalize more readily within Csmith than across program domains.

E. Autotuning CUDA Loop Nests

  • On GP100, tuning threading and inner-loop size reaches 73.5% of theoretical peak performance and matches PyTorch across tested problem sizes.The sweep also shows a performance drop near 100k threads.Source: E. Autotuning CUDA Loop Nests · p. 9

F. Learning a Cost Model using the State Transition Dataset

  • A Gated Graph Neural Network predicts instruction count with relative error 0.025, versus 1.393 for naive mean prediction.It is trained on 80% of the State Transition Database and validated on the remaining 20%; Figure 8 plots holdout performance across epochs.Source: F. Learning a Cost Model using the… · p. 9, Figure 8 · p. 9

G. Reinforcement Learning for LLVM Phase Ordering

  • Three of four reinforcement-learning algorithms achieve positive results on held-out Csmith programs, but only PPO scores positively on two of 13 other datasets.The agents are evaluated on random samples of 50 programs from each available dataset; the results highlight the challenge of cross-domain generalization.Source: G. Reinforcement Learning for LLVM… · pp. 9–10

H. Effect of Training Set on RL

Across the RL evaluations, training-set match and program representation relate to agent performance. The discussion presents CompilerGym as a unified platform for compiler-optimization research.

H. Effect of Training Set on RL

  • Agents performed best on benchmarks from their training dataset, suggesting the importance of training across a wide range of program domains.PPO agents were trained on different sets and evaluated on test sets from different domains.Source: H. Effect of Training Set on RL · p. 10

I. Effect of Program Representation on Learning

  • Adding a histogram of prior actions improved performance with both representations, and Autophase outperformed InstCount.Figure 9 plots holdout validation performance over training episodes for four PPO agents using different observation spaces.Source: §I · p. 10, Figure 9 · p. 10

VIII. RELATED WORK

  • CompilerGym combines compiler problems, representations, optimization targets, and offline datasets in one package, with larger search spaces than other compiler reinforcement-learning works.The paper also describes CompilerGym as a platform for supervised-learning data gathering and evaluation.Source: §VIII · p. 10
  • Prior LLVM phase-ordering work identifies generalization across programs as a key challenge; CompilerGym combines observation spaces with millions of training programs for research.Source: §VIII · p. 10

IX. CONCLUSIONS

  • CompilerGym aims to lower the entry barrier by removing the engineering investment required to try ideas on production compiler problems.Source: §IX · p. 10
Loading 2109.08267v2…