PaperlayerView exact revision on arXiv

Source-linked AI summary

Annual Interruption Rate as a KPI, its measurement and comparison

Rohit Pandey, Yingnong Dang, Ali Vira, Aerin Kim, Gil Lapid Shafriri, Murali Chintalapati

arXiv:1910.12200v1cs.DC

TL;DR

The paper addresses how Azure can quantify virtual-machine interruptions and compare failure rates despite stochastic observations and differing event processes. It defines AIR, discusses measurement from VM and heartbeat data, and applies hypothesis-testing ideas to reboot-count and inter-arrival models. The paper shows that test behavior depends on the underlying process, including reduced power for an inter-arrival Weibull process, while fixed-interval events have zero variance.

  • Problem

    Azure needs a normalized failure-rate KPI and statistically sound comparisons between groups when observed failure counts vary across finite, noisy data.

    Source: §1.1 · pp. 4–5

  • Method

    The paper defines AIR from an average event rate over VM-years, addresses interval-based VM downtime, and studies hypothesis tests under Poisson, compound Poisson, Weibull, and fixed-interval processes.

    Source: §1.1 · p. 5, Changing the inter-arrival distribution · pp. 15, 17, §2.3.1 · p. 18

  • Results

    The UMP Poisson test retains similar power on a Compound Poisson process but is less powerful on an inter-arrival Weibull process with κ = 0.5.

    Source: §2.3.2 · p. 21

  • Takeaways & Limitations

    AIR provides a single fleet- or scope-level interruption measure, while hypothesis-test thresholds may not equal actual false-positive rates when process assumptions are violated.

    Source: §1.1 · p. 5, §2.3.1 · p. 18

    Azure reboot counts are empirically over-dispersed, with variance much higher than the mean, challenging the Poisson-process assumption.

    Source: Hypothesis test for Poisson point… · p. 14

Abstract

from arXiv · show

This article is divided into two chapters. The first chapter describes the failure rate as a KPI and studies its properties. The second one goes over ways to compare this KPI across two groups using the concepts of statistical hypothesis testing. In section 1., we will motivate the failure rate as a KPI (in Azure, it is dubbed `Annual Interruption Rate' or AIR. In section 3, we will discuss measuring failure rate from logs machines typically generate. In section 1.2, we will discuss the problem of measuring it from real-world data. In section 2.1, we will discuss the general concepts of hypothesis testing. In section 2.2, we will go over some general count distributions for modeling Azure reboots. In section 2.3, we will go over some experiments on applying various hypothesis tests to simulated data. In section 2.4, we will discuss some applications of this work like using these statistical methods to catch regressions in failure rate and how long we need to let changes to the system `bake' before we are reasonably sure they didn't regress failure rate.

Introduction

The introduction frames failure rate as a measurable KPI for aligning Azure’s organization around improving virtual-machine availability and outlines the article’s measurement and hypothesis-testing chapters.

  • Azure uses KPIs to express virtual-machine availability goals numerically so progress can be tracked across the organization.Source: Introduction · p. 3
  • The article first studies failure rate as a KPI and then develops ways to compare it between groups using statistical hypothesis testing.Source: Introduction · p. 3
  • Annual Interruption Rate (AIR) names Azure’s failure-rate KPI, whose measurement is discussed alongside real-world data issues.Source: Introduction · p. 3
  • The second chapter covers hypothesis-testing concepts, reboot count distributions, simulated-test experiments, regression detection, and change-bake duration.Source: Introduction · p. 3

Annual Interruption Rate

The paper motivates Annual Interruption Rate (AIR) as a normalized failure-rate KPI aligned with interruption experience, then develops its measurement from machine logs and censored state-transition data.

1.1 Failure rate as a KPI

  • AIR replaces raw downtime and interruption counts with a normalized rate that supports comparisons across fleet sizes and activity levels.The KPI is based on projected events over VM-years, such as renting 100 VMs for one year.Source: §1.1 · pp. 4–5
  • Hazard rate represents the instantaneous event rate, while the KPI uses a single average rate computed as modeled events divided by the total observation interval.For an interval Δt, the average rate is λ = n/Δt.Source: §1.1 · pp. 4–5

1.2.1 Machine data

  • Machine logs are represented as heartbeat data or compressed state-transition data, with each state-transition row describing an UP or DOWN interval and possible censoring at processing-window boundaries.Heartbeat logs can be converted into state-transition data, which is more compact for analysis.Source: §1.2.1 · pp. 6–7

1.2.2 Some more metrics

  • For a two-state machine process, MTTF and MTTR summarize average UP and DOWN durations, but AIR excludes MTTR so prolonged downtime cannot make the failure KPI appear better.Under exponential assumptions, failure and recovery times are parameterized by rates λ and μ; the paper rejects a rate based on MTTR + MTTF for KPI use.Source: §1.2.2 · p. 7

1.2.3 Calculating MTTR and MTTF from heartbeat data

  • Censored and uncensored state-transition intervals can be treated alike when estimating MTTF, and λ becomes AIR when UP durations are measured in 100-VM-years.The approach avoids stitching censored observations across future processing windows.Source: §1.2.3 · p. 8

Comparing AIR between groups

Comparing AIR across groups requires hypothesis testing because observed failure-rate estimates are noisy. The framework balances false positives and false negatives, with data volume, effect size, and the underlying interruption process determining test sensitivity.

Comparing AIR between groups

  • System changes can regress failure rate, so deployments are compared by their observed failures per unit time against the pre-deployment state.The true rates remain unknown because finite, stochastic observations produce estimates with sampling variance.Source: Comparing AIR between groups · p. 9

2.1 Hypothesis testing

  • Hypothesis testing asks whether two populations differ in a statistical property despite noisy estimates, using a difference, ratio, or another test statistic.As more data are collected, the variance of the relevant estimate distributions is expected to decrease.Source: §2.1 · p. 10

The null hypothesis and false positive rate

  • Under the null hypothesis of equal AIRs, raising the rejection threshold lowers the false positive rate, but an infinite threshold makes the test useless because it never detects real differences.The threshold is defined through a controlled tail probability, often p=5%, rather than as a fixed test-statistic value.Source: The null hypothesis and false positive… · pp. 10–11

The alternate hypothesis and false negative rate

Choosing the best test

  • False positive and false negative rates trade off: shifting the threshold right improves false positives but worsens false negatives, while more data can improve false negatives without sacrificing the false positive rate.The best test depends on the stochastic process generating Azure VM reboots or other interruptions.Source: The alternate hypothesis and false… · p. 12, Choosing the best test · p. 13

2.2.1 The Poisson point process for modeling Azure VM reboots

The Poisson point process is a basic model for Azure interruption counts, treating reboots as point events when recovery times are negligible. Although it offers a uniformly most powerful comparison test, Azure reboot counts are empirically over-dispersed, challenging the model.

2.2.1 The Poisson point process for modeling Azure VM reboots

  • Azure can approximate interruptions as point events because mean time to failure is much greater than mean time to recovery.The underlying availability process is continuous-time and two-state, but down durations are ignored in this approximation.Source: §2.2.1 · p. 14
  • A Poisson process assumes exponentially distributed inter-arrival times with constant, memoryless hazard; counts over time t are Poisson with parameter λt.The parameter λ is both the event-rate parameter and the mean rate per unit time, while λt is also the count variance.Source: §2.2.1 · p. 14

Hypothesis test for Poisson point process

  • A uniformly most powerful test exists for comparing two Poisson processes and optimizes false negative rate for a specified false positive rate, effect size, and data amount.The resulting comparison reduces to a Binomial probability test implemented in Python and R.Source: Hypothesis test for Poisson point… · p. 14
  • Azure reboot counts are empirically over-dispersed, with variance consistently exceeding the mean, so the Poisson model may be unsuitable for hypothesis testing.Because variance is especially important to hypothesis tests, this mismatch is a potential deal-breaker and motivates modified count processes.Source: Hypothesis test for Poisson point… · pp. 14–15

2.2.2 Generalizations of the Poisson point process

The paper generalizes Poisson models to capture Azure failure counts whose variance may differ from their mean, using changing inter-arrival hazards and compound events.

2.3 Power of UMP Poisson test on Azure data

  • The generalized processes are evaluated by testing how the Poisson hypothesis test behaves when applied to simulated Azure-like data.The subsequent experiments assess the test on Compound Poisson and inter-arrival Weibull processes.Source: §2.2.2 · p. 15, §2.3 · p. 17

Changing the inter-arrival distribution

  • Azure failures can be clustered, so decreasing-hazard inter-arrival times create positive correlation and over-dispersed counts.After an event, the next-event hazard spikes and gradually declines, increasing nearby arrivals and variance.Source: Changing the inter-arrival distribution · p. 15
  • The Weibull distribution models both increasing and decreasing hazard rates, and its resulting count process can be simulated even without a known closed-form PMF or CDF.Simulation uses inverse-transformed Weibull inter-arrivals and counts arrivals within each observation period.Source: Changing the inter-arrival distribution · pp. 15, 17
  • Increasing-hazard inter-arrival times push events apart, producing negative correlation and under-dispersed counts.An extreme increasing-hazard process produces periodic events with zero variance.Source: Changing the inter-arrival distribution · pp. 15, 17

The Compound Poisson Process

  • The Compound Poisson process models node failures that bring down random numbers of co-hosted VMs, allowing count variance to exceed the mean.A Poisson node-down event is assigned a random number of VM interruptions, producing over-dispersion.Source: The Compound Poisson Process · p. 17

2.3.1 UMP Poisson test on Compound Poisson process

Applying the UMP Poisson test to Compound Poisson data preserves comparable power but distorts the relationship between the configured and actual false-positive rates.

2.3.1 UMP Poisson test on Compound Poisson process

  • The UMP Poisson test has α = ˆα on Poisson data, but Compound Poisson data requires calibrating ˆα because the actual α differs.Simulation shows the Compound Poisson mapping rises rapidly toward 0.3, then 0.6, before reaching 1 as ˆα goes from 0 to 1.Source: §2.3.1 · p. 18, Figure 2.7 · p. 19
  • The α-to-ˆα mapping remains stable across tested observation sizes and base rates, allowing one lookup mapping for compared Compound Poisson groups.The authors report this stability empirically rather than from a rigorous proof.Source: §2.3.1 · p. 18, Figure 2.9 · p. 20
  • The test’s α-β curves for Poisson and Compound Poisson processes are nearly overlapping, indicating comparable power after false-positive calibration.The distribution mismatch changes the actual false-positive rate, but the reported power remains similar.Source: §2.3.1 · p. 19, Figure 2.8 · p. 20

2.3.2 UMP Poisson test on inter-arrival Weibull

The UMP Poisson test is less powerful for decreasing-hazard Weibull arrivals, while scaling counts changes false-positive calibration without improving the α-β relationship.

2.3.2 UMP Poisson test on inter-arrival Weibull

  • The UMP Poisson test has a higher false-negative rate and lower power on the κ = 0.5 inter-arrival Weibull process than on Poisson or Compound Poisson data.In figure 2.8, the Weibull curve lies above the Poisson and Compound Poisson curves.Source: §2.3.2 · p. 21
  • Scaling both groups’ event counts by 25 or 0.1 changes the relationship between configured ˆα and actual α but leaves the α-β relationship unchanged.The scaled curves overlap the unscaled Weibull curve, so the modification does not change test power.Source: §2.3.2 · p. 21

2.4.1 Theoretically sound hypothesis test for AIR

  • For Azure’s likely over-dispersion, the paper argues that compounding may be more plausible than hazard-rate effects and supports using the UMP Poisson test with an α lookup table.The proposed construction matches Azure’s mean and variance and is intended to provide close to the best possible power.Source: §2.4.1 · pp. 21–22
  • The calibrated AIR hypothesis test has been implemented in Kusto and is used by Azure teams to compare failure rates.The implementation follows the simulated mapping between desired and configured false-positive rates.Source: §2.4.1 · p. 22

2.4.2 Recommending time to wait

The simulator converts desired effect size and error rates into observation-window recommendations for deployment comparisons.

2.4.2 Recommending time to wait

  • The simulator obtains a mapping between desired α and the configured ˆα needed to achieve the target false-positive rate.This mapping is one of the simulator’s stated purposes.Source: §2.4.2 · p. 22
  • Observation durations t1 and t2 can be selected by simulating until a chosen effect size and target false-positive and false-negative rates are satisfied.With one interval constrained, the other can be varied across the t1-t2 space until the requirements are met.Source: §2.4.2 · p. 22
  • The simulator should use the same hypothesis test intended for issue detection and can use a Poisson model because its UMP test has equivalent power on Compound Poisson data.For over-dispersed data, the Poisson-based waiting time is treated as a conservative lower bound relative to real-data waiting time.Source: §2.4.2 · p. 22
  • An online simulator is used to determine how long to wait between deployment stages.The tool operationalizes the observation-window calculation for deployment decisions.Source: §2.4.2 · p. 22

Derivation of hazard rate function

The section develops hazard rate as an instantaneous event rate conditioned on survival to time t, then shows how to estimate an average rate over an interval from observed event counts and exposure time.

Derivation of hazard rate function

  • The derivation begins with the inter-arrival time T and its density f_T(t), which describe when successive events occur.The conditional probability of an event immediately after t is related to the density of T.Source: Derivation of hazard rate function · p. 24
  • The hazard rate is the instantaneous rate of an event occurring after no event has occurred by time t.It answers how many events are expected per unit time conditional on survival until t.Source: Derivation of hazard rate function · p. 24
  • The hazard rate h_T(t) represents the expected number of events in a sufficiently small interval as h_T(t)δt.For sufficiently small intervals, the event count is effectively Bernoulli, so its probability equals its expected count.Source: Derivation of hazard rate function · p. 25
  • For a time interval, the average rate λ is estimated as the observed number of events divided by the interval’s total duration.This follows the same total-distance-over-total-time logic used to define average velocity.Source: Derivation of hazard rate function · p. 25

Using the exponential distribution to estimate AIR

The section uses the exponential distribution because its constant failure rate supports a single-rate estimate, deriving MTTF from censored and uncensored observations and relating AIR to total failures over total UP time.

Using the exponential distribution to estimate AIR

  • The exponential distribution is the only distribution with a constant failure rate, making it suitable for estimating a single failure rate.The method applies when observations include both censored and uncensored data.Source: Using the exponential distribution to… · p. 26
  • The exponential model’s likelihood combines uncensored failure times with censored observations before optimization determines λ.The derivation uses the exponential PDF, survival function, log likelihood, and a zero-derivative condition.Source: Using the exponential distribution to… · pp. 26–27
  • Summing all censored and uncensored UP times and dividing by downtime events yields the MTTF; the analogous calculation gives MTTR from DOWN times.The estimator is obtained through the exponential likelihood and its derivative with respect to λ.Source: Using the exponential distribution to… · pp. 26–27
  • The AIR estimate is 1/MTBF, equivalently the total number of failures divided by total UP time.This expresses the failure-rate KPI using accumulated operating exposure.Source: Using the exponential distribution to… · p. 27

Related papers

Loading 1910.12200v1…