Protean Compiler: An Agile Framework to Drive Fine-grain Phase Ordering

1 Introduction

• Pass ordering is open-ended: one optimization can cancel another’s benefits or create opportunities for later passes.The paper describes the phase-ordering problem as open for more than 45 years.Source: §1 · p. 2
• Existing ML-enabled compiler approaches target selected passes, but do not guide the pass manager’s end-to-end ordering of optimization phases.Protean contrasts this limitation with its fully integrated approach.Source: §1 · p. 2
• Protean integrates LLVM phase ordering at fine-grained scopes, with an agile optimizer seeking an effective pass ordering for each scope.The framework accommodates scopes such as modules, call graphs, functions, and loops.Source: §1 · p. 2
• Protean includes IR2Score, a Transformer-based performance model, and Protean Feature Set collectors for evaluating recipes across scopes.The framework also supports third-party features such as IR2VEC.Source: §1 · p. 2
• Protean supports integration with third-party ML frameworks and LLMs for optimization use cases.The paper presents this flexibility as part of the framework’s capabilities.Source: §1 · p. 2

2 Related Work

• Phase ordering has no ideal order, is unbounded, and remains NP-hard even when bounded; scarce labeled data also limits reuse and reproducibility.Some prior work therefore uses alternative objectives such as code-size reduction.Source: §2 · p. 3
• CompilerGym [12] wraps compiler optimization for phase-ordering experiments, including code-size optimization, whereas Protean targets running-application performance within LLVM.The paper also notes CompilerGym’s use of reinforcement learning techniques.Source: §2 · p. 3
• MiCOMP uses predictive models for iterative LLVM compilation but is a Python wrapper, operates program-wide, and uses dynamic features.Protean contrasts this with finer scopes and static features or pretrained embeddings at inference.Source: §2 · p. 3
• Protean distinguishes itself from MLGO, MLGOPerf, and ACPO by guiding pass ordering in a fully integrated compiler at finer granularity.Its stated novelty combines fine-grained scope with a tailored phase-ordering recipe for a code segment.Source: §2 · p. 3

3 Proposed Framework

• The pipeline converts source to LLVM IR, partitions the program, optimizes each partition iteratively, and links the resulting outputs.Figure 1 is identified as the Protean Compiler System Architecture.Source: §3 · p. 4, Figure 1 · p. 5
• Simulated annealing drives recipe search, while IR2Score evaluates the resulting LLVM IR and assesses solution quality.Recipes are built from available pass subsequences; performance optimization treats a higher score as predicted improvement.Source: §3 · p. 4
• The acceptance cost compares a newly tested recipe with the current recipe and depends on the temperature setting.The passage defines C_new as the new recipe’s cost, C_present as the present recipe’s cost, and T as temperature.Source: §3 · p. 4
• High temperature permits poorer solutions during broader exploration; lowering it reduces their acceptance until the search refines a recipe for each partition.Optimized partitions are linked, with a further optimization step if LTO is used.Source: §3 · p. 4

3.1 Agile Driver

• The -OP flag adds Protean as an optional Clang optimization level that coexists with existing optimization flows.The implementation adds a ProteanOpt phase and a new compilation executable for the agile loop.Source: §3.1.1 · p. 5, §3.1 · p. 5

3.1.1 Clang Driver Modifications.

• The protean tool replaces a fixed pipeline with an iterative loop, rejecting recipes whose subprocess exits abnormally and continuing the search.The child process applies a recipe to a module before IR2Score evaluates its resulting IR.Source: §3.1.1 · p. 5

3.2 Supporting Parallel Builds

• Concurrent module explorations use separate shared-memory buffers for feature collection, avoiding interference and additional feature-collection disk I/O.The buffers support both PFS and IR2VEC feature collection techniques.Source: §3.2 · p. 6

3.3 Protean Recommender System

• The recommender uses genetic selection, crossover, and mutation to explore recipes, while simulated annealing contributes local refinement.The paper presents the hybrid as balancing broad search with hill-climbing.Source: §3.3 · p. 6

3.4 Protean Subsequences

• Shorter subsequences reduce the optimization space, while their fixed internal pass order preserves intended dependencies such as inlining before loop unrolling and vectorization.Inlining can expose opportunities for later optimizations.Source: §3.4 · p. 6

3.4.1 Optimization Clustering.

• Agglomerative clustering uses a weighted optimization-dependency graph derived from consecutive pass pairs in -O3 to produce five subsequences, A through E.The graph weights each pair by how often the passes occur consecutively in -O3.Source: §3.4.1 · p. 7
• A maximum recipe length of 5 reaches 97% of achievable speedup while reducing the search space by orders of magnitude relative to length 7.The choice was evaluated on representative applications from six CBench categories.Source: §3.4.2 · p. 7

• The model assumes that applying an optimization-pass set together is equivalent to applying its passes one by one in order, relating final-code features to features from earlier pass orders.Here, o[i] denotes the optimization pass at order i.Source: §3.5 · p. 7
• A Transformer models sequential relations between IR states and reports higher accuracy and lower validation loss than an MLP-like fully connected network.The representation uses the feature map while preserving prior IR-state features in the learning process.Source: §3.5 · pp. 8–9, Figure 2 · p. 9
• IR2Score takes LLVM IR features as input and predicts speedup relative to the -O3 baseline for a given subprogram.It can also accept sequential feature states alongside the current state when available.Source: §3.5 · pp. 7, 9

• PFS contains 141 handcrafted features spanning module, call-graph, function, and loop scopes, with group-specific collectors for collecting features efficiently.Feature groups include examples such as function counts, call-graph metrics, and loop sizes and trip counts.Source: §3.6 · pp. 9–10
• PFS is registered at the end of the agile compilation pipeline to provide updated features from each transformed IR state for model inference.The feature library or individual scopes can be added to IR2Score training and inference.Source: §3.6 · p. 9
• IR2Vec has higher average correlation across its 300 embeddings, while PFS has a dozen strongly correlated features with the speedup label.The comparison reports Pearson correlations for 141 PFS features and 300 IR2Vec features.Source: §3.6 · p. 10

4 Experimental Results

• The evaluation uses module-level optimization, single-threaded runtime measurements, and repeated benchmark runs, with transformed IR evaluated iteratively by IR2Score.Runtime measurements use Linux perf; runs with more than 1% variance are repeated.Source: §4 · pp. 10–11
• IR2Score is trained on 12,000 C/C++ functions, and weighted mean-square loss improves accuracy for samples below 0.8 or above 1.5 speedup.The reported cross-validation loss for the Transformer is 0.094, versus 0.958 for MLPs and 0.125 for LSTMs.Source: §4 · p. 11, Figure 4 · p. 13
• Adding the genetic algorithm to simulated annealing moves the search toward better speedups earlier, with the largest performance gains occurring between 10 and 50 iterations.The genetic algorithm starts at iteration 20.Source: §4 · pp. 11–12, §4.1.1 · p. 12

4.1 Performance Analysis of Protean Compiler

• CBench speedups range from 2.2% to 4.1% on average, with select applications reaching up to 15%; PFS consistently outperforms IR2VEC.With 10 to 50 iterations, average speedup rises from 2.2% to 3.3% for IR2VEC and from 2.5% to 3.5% for PFS.Source: §4.1.1 · p. 12, Table 4 · p. 14

4.1.2 SPEC CPU 2017.

• Protean improves SPEC CPU 2017 geometric-mean performance by 1% in 1-copy runs and 1.1% in 64-copy runs, with a 1.5% code-size increase.Benchmark 557.xz improves by 1.3% in 1-copy and 6.5% in 64-copy runs.Source: §4.1.2 · p. 14, Table 5 · p. 15

4.2.1 CBench.

• Early stopping limits CBench build-time overhead; with PFS, benchmarks averaged about 145 iterations, compared with about 310 for IR2VEC.The unchanged-IR early-exit condition is set to 100 iterations, and overhead rises little for PFS between 100 and 500 configured iterations.Source: §4.2.1 · pp. 14–15, Table 6 · p. 16

4.2.2 SPEC CPU 2017.

• Parallel builds benefit Protean more than -O3 for some SPEC workloads because sequential Full LTO can dominate total build time while Protean’s search runs at the IR stage.The passage identifies 500.perlbench, 502-gcc, and 523-xalancbmk as examples where the sequential task can dominate.Source: §4.2.2 · pp. 15–16

5 Comparative Results

• The comparison accounts for MiCOMP’s LLVM 3.8 pass sequences and unsupported recent AArch64 tooling, so it uses a proxy rather than a direct reproduction.Protean uses LLVM 19.Source: §5 · p. 16
• Using module-level Protean results as a proxy for MiCOMP’s program-level scope, Protean reports speedups over MiCOMP of 1.6%, 2%, 1.4%, and 1.1% at 10, 50, 100, and 500 iterations, respectively.Source: §5 · pp. 17–18
• On CBench, MiCOMP’s 1-shot Top-1 prediction averages 40–45 seconds, roughly matching Protean’s 10-iteration PFS overhead on its single-core compilation approach.At equal iteration counts, Protean’s compile overhead is an order of magnitude lower than MiCOMP’s.Source: §5 · p. 18

6 Integration Capabilities

• The framework can integrate third-party ML frameworks for pass-parameter selection and combine LLM code generation with Protean phase ordering.The examples use 20 Protean iterations and IR2Score trained with IR2VEC features.Source: §6 · p. 19

6.1 Parameter Tuning with Third-party ML Frameworks

• Protean selects pass subsequences but leaves their internal parameters to default O3 heuristics; parameter selection is a separate, complementary problem.The example pairs Protean's recipe selection with ACPO's Function Inlining model, which decides whether to inline callsites.Source: §6.1 · pp. 19–20, §6 · p. 19
• 8.5% speedup over -O3 on consumer_jpeg_c came with a 12% binary-size increase; Protean plus ACPO produced a 5.38-second runtime.The ACPO Function Inlining model makes callsite decisions across 55 modules when the selected recipe includes Inliner.Source: §6.1 · p. 20

6.2 Code Optimization with LLM Integrations

• The LLM workflow generates function-preserving source-to-source candidates, validates correctness, then compares Protean compilation against compiling the same rewritten source.The authors present one full example and leave a fully automated LLM optimization framework for future work, citing automation, verification, and reliability challenges.Source: §6.2 · p. 20
• A revised Susan test case gained 3.1% speedup from the LLM rewrite over -O3 and a further 7% from Protean over the LLM-optimized source.Protean's recipe CD was the best solution found at iteration 11; its module speedup prediction was around 8%.Source: §6.2 · pp. 21, 23

7 Discussion & Conclusion

• Protean integrates feature embeddings, ML frameworks, and LLM-based code optimization, with automation, verification, and reliability challenges remaining for LLM integration.The reported examples combine optimization at algorithmic and fine-grained levels.Source: §7 · p. 23
• The feature library contains handcrafted features at different program scopes, already integrates IR2VEC, and is designed to accommodate other feature methods.The authors plan to extend the library and release improved IR2Score models.Source: §7 · p. 23
• Protean's framework supports program-, module-, function-, and loop-level optimization, although this work showcases module-level scope.The authors say the framework and feature set allow users to build strategies at other granularities.Source: §7 · p. 23
• Protean reports up to 4.1% average speedup and up to 15.7% on selected CBench applications, plus 1% average and up to 6.4% on selected SPEC CPU 2017 Integer applications.The conclusion also reports integration results of 10.1% on Susan and 8.5% on Jpeg.Source: §7 · pp. 23–24

A.1 Protean Feature Set (PFS)

• For a one-function, one-loop module, PFS produces two scope-specific feature rows and can aggregate features for larger modules.The example has module-level features in one row and function- and loop-level features in the other.Source: §A.1 · p. 27

A.2 Protean Compiler Hyperparameters

• The listed simulated-annealing settings include cooling schedule, maximum iterations, random seed, maximum temperature, and initial sample size.The available cooling choices are geometric and linear.Source: §A.2 · pp. 27, 29
• The genetic recommender exposes mutation and crossover rates, population size, crossover type, and mutation type.Listed crossover options include single-point, double-point, and uniform; mutation options include FlipOne and SwapTwo.Source: §A.2 · p. 29
• The simulated-annealing cost can use IR2Score-based IR analysis or llvm-mca cycle counts.The appendix also lists an option to use Protean-collected features for the IR analyzer.Source: §A.2 · p. 29