Virtual Machine Warmup Blows Hot and Cold

1 INTRODUCTION

• The study tests whether small, deterministic programs reach a steady state of peak performance, a premise behind discarding measurements collected before warmup.The authors deliberately use a highly idealised setting to test this hypothesis.Source: §1 · p. 2
• Only 30.0–43.5% of ⟨VM, benchmark⟩ pairs consistently reach a steady state of peak performance, and none of the seven studied VMs does so consistently.The range depends on the machine and OS combination.Source: §1 · p. 2

1.1 Overview of the Methodology

• The experiment runs benchmarks for 2000 in-process iterations across 30 fresh process executions, using Krun to control practical sources of measurement variation.It also tests different machines and operating systems.Source: §1.1 · p. 2
• Automated changepoint analysis classifies each execution as no steady state, flat, warmup, or slowdown, revealing patterns that previous methodologies cannot tease apart.The analysis detects shifts in time-series behavior and supports determining whether and when a steady state is reached.Source: §1.1 · p. 2

1.2 Summary

• The paper contributes an automated method for detecting steady states and classifying flat, warmup, and slowdown behavior, alongside evidence that studied benchmarks often fail to reach peak steady state.The authors also report that many unexpected effects are not explained by GC or JIT compilation events; the approach is open-source and applicable to VM benchmarking data.Source: §1.2 · p. 3

2 BACKGROUND

• JIT VMs are traditionally understood to interpret programs initially, then compile frequently executed code into machine code, after which execution is considered to have reached peak-performance steady state.The length of warmup depends on both the program and the JIT compiler.Source: §2 · p. 4
• Common benchmarking practice discards an initial fixed number of iterations, often around five, without guaranteeing that warmup has finished.More advanced methods use simple statistical tests to seek a steady state.Source: §2 · p. 4
• Kalibera and Jones’s manual method uses initial process executions to identify a steady-state cutoff, then applies it to more executions for statistical analysis.Manual identification is error-prone and time-consuming, limiting its practical use to a small number of initial executions; the cutoff is then applied without assurance that later executions reach steady state at that point.Source: §2 · p. 4
• The manual method does not otherwise classify steady-state performance relative to earlier execution, making it difficult to tell whether steady state represents peak performance.The paper frames rigorous identification of warmup and its definition as an open research problem.Source: §2 · p. 4

3 METHODOLOGY

• The experiment runs a suite of microbenchmarks for 2000 in-process iterations across 30 process executions, with controls intended to support repeatability.The methodology covers benchmarks, VMs, hardware, and the Krun system.Source: §3 · p. 5

3.1 The Microbenchmarks

• The suite uses small, widely studied microbenchmarks because their deterministic and well-examined behavior makes them a reliable test of the hypothesis.The benchmarks include binary trees, spectralnorm, n-body, fasta, fannkuch redux, and Richards.Source: §3.1 · p. 5

3.1.1 Ensuring Determinism.

• The authors check user-level determinism by comparing branch-point outputs across runs, fixing fasta’s unreseeded random generator and an observed Java class-loading variation.They do not attempt to control nondeterminism within the VM, which is part of what they test.Source: §3.1.1 · p. 5

3.2 Measuring Computation Rather than File Performance

• Per-iteration checksums discourage optimizers from removing benchmark computation without including file-output performance in the measurement.The same expected checksum across language implementations also provides assurance that they perform roughly the same work.Source: §3.2 · p. 6

3.3 VMs under Investigation

• The study uses fixed, repeatably built versions of GCC and seven JIT VMs, while omitting specific VM–platform and benchmark combinations that were unavailable, crashed, or took impractically long.The omitted cases include three VMs on OpenBSD, fasta on JRuby+Truffle, and Richards on HHVM.Source: §3.3 · p. 6

3.4 Benchmarking Hardware

• Three machines vary hardware while holding Debian constant, or vary operating system while holding hardware constant, to test the effect of these differences.The hardware and OS comparisons are framed as moderately different configurations.Source: §3.4 · p. 6

3.5 Krun

• Krun supervises benchmark execution and data collection, rebooting before each process execution and controlling machine temperature and other practical sources of interference.It also manages a fixed-size schedule and prepares fresh benchmark-user accounts; the authors disabled daemons and network interfaces where possible.Source: §3.5 · p. 7, §3.5.1 · p. 7

3.5.2 Linux-specific Controls.

• Krun reduces platform-specific interference by setting Linux CPU and kernel controls, limiting perf sampling, and setting OpenBSD CPU performance and malloc options.The Linux controls include performance mode and a tickless kernel; on OpenBSD, Krun invokes apm -H and uses malloc.conf flags cfgrux.Source: §3.5.2 · p. 8, §3.5.3 · p. 8

4 CLASSIFYING WARMUP

• The method analyzes 2,000-iteration timing series to classify warmup automatically, using outlier filtering and changepoint analysis of shifts in mean and variance.The PELT algorithm reduces changepoint search complexity to O(n); the analysis uses a 15 log n penalty.Source: §4 · p. 9, §4.1 · p. 10, §4.2 · p. 10

4.1 Outliers

• Within sliding windows of 200 iterations, values beyond the median ±3 × (90%ile −10%ile) are outliers; the first 200 iterations are exempt to retain likely warmup data.Across 7,320,000 iterations, 0.3% were classified as outliers; the maximum for one execution was 11.2%.Source: §4.1 · p. 10

4.2 Changepoint Analysis

• Changepoint analysis detects shifts in time-series properties, avoiding manual or simple-heuristic judgments about when iteration timings change.The method uses PELT and identifies changes in both mean and variance; the penalty permits an independence assumption despite observed dependence in some data.Source: §4.2 · p. 10

4.3 Classifications

• Changepoint analysis cannot identify the domain’s noise floor or guarantee that similar segments separated by a distinct segment will be treated as equivalent.Source: §4.3 · p. 10
• An execution is steady-state when the final 500 iterations are equivalent to the final segment; executions and VM–benchmark pairs are then classified by their segment patterns and cross-run consistency.Equivalence uses the final segment’s mean ± max(variance, 0.001s). Steady-state labels are flat, slowdown, or warmup; pairs with differing execution labels are inconsistent, while no steady state and slowdown are classified as bad outcomes.Source: §4.3 · pp. 11–12

4.4 Steady State Timings

• For steady-state executions, the analysis reports iterations and seconds to steady state plus steady-state performance, using IQRs for timing spread and bootstrapped 99% confidence intervals for performance.IQRs report the median and 5th and 95th percentiles; non-overlap implies a meaningful timing difference, though wide spreads may prevent one from being established. Performance intervals use 100,000 bootstrap iterations, sampling within but not across segments.Source: §4.4 · p. 12

5 RESULTS

• 43.4% of Linux1240v5 VM–benchmark pairs had consistently good warmup, while 43.5% were bad inconsistent; Linux4790 had similar proportions, but OpenBSD4790 differed.Table 1 summarizes classifier proportions across machines; OpenBSD4790’s bad-inconsistent share was 70.0%, partly because it lacked JRuby+Truffle, the most consistent VM on Linux.Source: §5 · pp. 12–13, Table 1 · p. 13
• On Linux1240v5, n-body and spectralnorm came closest to good warmup across VMs, while binary trees, fasta, and Richards often showed bad inconsistency.Steady states were reached either quickly, often within 10 in-process iterations, or only after hundreds; longer warmups did not appear to yield faster steady-state performance.Source: §5 · p. 13, Table 2 · p. 14

5.1 Warmup Plots

• Run-sequence and per-core cycle plots show benchmarks migrating between cores, sometimes alongside changepoints; continual changepoints can produce a no-steady-state classification.Core-cycle plots help reveal VM thread use and OS scheduling; several cores can be active during early iterations when JIT compilation occurs.Source: Figure 2 · p. 15, §5.1 · pp. 15–17
• The analysis captures cycles in wall-clock time, but their performance changes can be too small for the classifier to label the run no steady state.The plots also include examples of bad inconsistency (one execution warming up and another slowing down) and good inconsistency (one classified as warmup and another flat).Source: Figure 3 · p. 15, Figure 4 · p. 15, Figure 5 · p. 16

5.2 The Effects of Compilation and GC

• Some non-warmups correlate with GC or JIT events, but many do not, leaving the authors unable to validate or invalidate their hypothesis that these events largely explain non-warmup behavior.The authors report relatively few instrumented results, and the instrumentation was available only for HotSpot and PyPy among the VMs considered.Source: §5.2 · p. 17

6 STARTUP TIME

• Startup time is the interval from just before loading the VM to the first point when user-level code can execute.A wrapper records wall-clock time before executing the VM, and a dummy runner records it when user-level execution begins.Source: §6 · pp. 17–18
• Each VM’s startup time was measured over 200 process executions, with Krun rebooting before each run.Because in-process iterations are irrelevant for these runs, the dummy program completes as soon as it prints the wall-clock time.Source: §6 · p. 18
• With C on Linux as the baseline, LuaJIT started 5× slower and JRuby+Truffle around 3500× slower.The measurements are cold starts and partly reflect disk speed, although the machines load VMs from an SSD.Source: §6 · p. 18

7 THREATS TO VALIDITY

• The study cannot account for unknown confounding variables that may have colored its results.Hardware comparisons across distinct architectures can also change the VM’s JIT components, making apples-to-apples comparisons often impossible.Source: §7 · p. 18
• The study did not systematically test whether rebuilding VMs affects warmup, and checksums may make benchmarks unrepresentative.Checksums ensure equivalent work at a user-visible level, but not necessarily at the lowest level; rebuilding on the machines provides only limited evidence about rebuild effects.Source: §7 · p. 18
• The benchmark runners cannot avoid all system calls; clock_gettime has no guaranteed return-time bound, although it ran faster than benchmark granularity in practice.Reading MSR device nodes adds file-system overhead and triggers inter-processor interrupts; the authors know of no practical way to lower these costs.Source: §7 · pp. 18–19
• Modern CPUs may not follow operating-system frequency requests, and APERF/MPERF readings have measurement error.The authors report around 1% error on active cores, while idle-core error can be 10% or more; their checks found frequency violations in 0.004% of iterations.Source: §7 · p. 19
• CPU shielding worsened performance for VMs such as HotSpot, possibly because fewer visible cores made compilation threads compete for core resources.The experiments therefore allowed the kernel to run on the benchmarking core.Source: §7 · p. 19
• Thirty process executions reduce, but do not eliminate, the chance that measurements sample only a narrow portion of the ASLR search space.The authors could not build Stabilizer on a modern Linux system and left ASLR enabled on Linux to ease comparison with OpenBSD.Source: §7 · p. 20
• The controlled environment deviates from standard configurations and may produce warmup behavior few users encounter in practice.The authors judge this a reasonable trade-off against substantial measurement noise.Source: §7 · p. 20
• Changepoint analysis assumes independence within segments despite dependence in 11.8% of process executions; the authors report that resulting confidence intervals tend to over-approximate.They used a larger penalty than typical for their setup and report that manual inspection generally found the automatic changepoints no worse than their initial attempts.Source: §7 · p. 20

8 RELATED WORK

• The study strengthens Gil et al.’s observations of inconsistent, no-steady-state, and slowdown runs by testing more benchmarks and VMs under tighter control.It also adds an automated approach for identifying warmup and classifying run-sequence plots.Source: §8 · pp. 20–21
• Changepoint analysis classifies large cycles as no steady state, while small cycles may yield equivalent-mean segments or a single segment.This contrasts with Kalibera and Jones’s manual selection of one part of a cycle for bootstrapping, which can introduce bias toward its slow or fast part.Source: §8 · p. 21
• 78.1% of process executions classified as no steady state were labeled steady state by Georges et al.’s coefficient-of-variation heuristic.The heuristic used a threshold of 0.01 and confirms that simple approaches can give misleading VM benchmarking statistics.Source: §8 · p. 21

9 SUGGESTIONS FOR THE COMMUNITY

• Each process execution should be analyzed individually: runs of the same VM–benchmark pair can reach steady state at different times or never reach it.Discarding a large fixed number of initial iterations does not guarantee that the remainder represents steady state.Source: §9 · p. 21
• Steady-state-only reports can hide large warmup differences: VMs within 2× in steady-state performance may differ by 100–1000× in warmup.The authors recommend reporting the benchmark classification, when steady state was reached, and steady-state performance.Source: §9 · pp. 21–22
• They recommend about 0.5 seconds for the fastest in-process iterations, or at least 0.1 seconds when that target is impractical.Very short measurements can be disproportionately affected by external events such as context switches.Source: §9 · p. 22
• They suspect small benchmark sets can over-train VM heuristics and miss VM behavior, while larger suites are harder to assemble when candidate benchmarks are nondeterministic.They suggest separate training and validation benchmark sets for heuristic development and evaluation.Source: §9 · p. 22

9.1 How Long Should Experiments Be Run For?

• Traditional experiments often use 5 or 10 in-process iterations, but the authors found such short runs insufficient for statistically robust analysis.Their experiment used 2000 iterations, chosen to extend substantially beyond VM behavior changes they expected around 1000 iterations.Source: §9.1 · p. 22, §9.1.1 · p. 23

9.1.1 In-process iterations.

• JRuby+Truffle slowdowns around iteration 1050 can make truncated runs classify an execution as having no steady state when the transient segment appears.At n = 1000 the slowdown is missed, while at n = 1050 a new changepoint can alter the classification.Source: §9.1.1 · p. 24, Figure 10 · p. 24
• Overall similarity was 85% at n = 700 but 78% at n = 900, making a safe iteration count difficult to choose across VMs.The authors suggest using around 500 iterations in many cases and occasionally 1500 to check longer-term stability.Source: §9.1.1 · p. 24

9.1.2 Process executions.

• With 10 process executions, results would have been very similar to the final experiment, and with 15 they would have been almost identical; two executions gave relatively poor results.The authors observed the beginning of diminishing returns at six executions and note that occasionally using more can help identify infrequent performance issues.Source: §9.1.2 · p. 25

10 CONCLUSIONS

• The authors report that their carefully designed experiment invalidated the restricted version of the informal warmup definition captured in Hypothesis H1.The analysis techniques are open source for VM developers and users to investigate warmup behavior.Source: §10 · p. 25
• The authors’ experiment took around 3 person-years, partly because detailed experiments had little precedent and required investigating and controlling confounding variables.They note that few researchers may have the time or energy to institute all the controls used.Source: §10 · pp. 25–26
• Which experimental controls matter most remains an open question, and untangling their partly interlocking effects may require many months of additional running time.Source: §10 · p. 26

A APPLYING THE STATISTICAL METHOD TO EXISTING BENCHMARK SUITES

• The method was applied using 30 process executions and 2000 in-process iterations; DaCapo ran only on Linux4790, while Octane ran across three machines.DaCapo used its default benchmark size; Octane included 12 benchmarks, including lightly modified pdfjs and typescript.Source: A APPLYING THE STATISTICAL METHOD TO… · p. 28
• Only 42% of DaCapo ⟨VM, benchmark⟩ pairs had “good” warmup.The comparison with HotSpot is difficult because Graal could run only a subset of benchmarks; among benchmarks both ran, results showed reasonable similarity.Source: A APPLYING THE STATISTICAL METHOD TO… · p. 28, Table 4 · p. 29
• Across three machines, 33% of Octane ⟨VM, benchmark⟩ pairs had “good” warmup, with a greater spread of classifications than DaCapo.Source: A APPLYING THE STATISTICAL METHOD TO… · p. 28, Table 5 · p. 30, Table 6 · p. 31
• The authors cannot determine whether larger benchmark codebases make “good” warmup more or less likely, and note that code-path nondeterminism may affect results.Octane’s modified runner could not fully replicate the original benchmark order, which could affect later benchmarks in the same process.Source: A APPLYING THE STATISTICAL METHOD TO… · p. 28

B FURTHER RESULTS

• Tables 8 and 9 present benchmark results for Linux4790 and OpenBSD4790, respectively.Source: B FURTHER RESULTS · p. 33, Table 8 · p. 34, Table 9 · p. 35

• Four interesting plots from each classification are selected to illustrate the range of experimental data.Source: C CURATED PLOTS · p. 36
• The appendix contains curated plots rather than the complete series.Source: C CURATED PLOTS · p. 36
• A separate document contains the complete series of plots.Source: C CURATED PLOTS · p. 36