Source-linked AI summary
Tremulant: Latency in Serverless Platforms beyond the 99th Percentile
Linus Gustafsson, Trever Schirmer, David Bermbach
TL;DR
Cold starts have drawn mitigation efforts, but the extreme FaaS latency tail also includes slow warm starts. The paper analyzes warm-tail behavior across datasets and finds that warm starts contribute to extreme latencies, while FaaS remains less consistent than comparable IaaS.
Problem
Method
The study analyzes warm-tail behavior in a primary FaaS dataset and validates its findings using additional datasets made tail-comparable.
Source: §6.1 · p. 6, §7 · p. 9Results
Across the primary and additional datasets, warm starts contribute to extreme tails, and all evaluated FaaS platforms perform considerably worse than comparable IaaS.
Source: §7 · p. 9Takeaways & Limitations
As cold-start impact decreases, improving perceived FaaS latency consistency increasingly requires understanding and mitigating warm-start tail events.
Source: §8 · p. 10The analysis isolates tail events using empty functions or calculated platform delay; real workloads have longer runtimes that reduce their relative impact.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Function-as-a-Service (FaaS) increases developer productivity by removing operational concerns such as managing hardware and software runtimes. Because function instances are started on demand, some requests must wait for a new instance, called a cold start. These are a common obstacle to more widespread adoption of FaaS, as they lead to unpredictable latency spikes that are not acceptable for some use cases. To solve this, many cold start optimizations have been implemented. However, comparatively little attention has been directed towards the slowest warm starts. With improvements in cold start times, slow warm starts emerge as a new problem for extreme tail events. By analyzing a dataset of several million requests across seven FaaS platforms, we show that the long tail of slow warm starts can be hundreds of times slower than their median performance, while a virtual machine we use as a baseline is substantially more consistent. We confirm this across four independent public datasets spanning 2020 to 2025, so the effect is neither new nor specific to one platform. Moreover, the warm share among the slowest calls starts to rise again at the highest percentiles: the most extreme latencies are not only cold starts, and cold-start mitigation alone cannot remove them.
1 Introduction
FaaS latency consistency matters for user-facing quality and execution cost, but research has focused more on cold starts than on slow warm invocations. The paper examines warm-start tails across platforms and public datasets.
Motivation
- At p99.99, warm starts are 6.5 to 947.5 times slower than their median, while AWS EC2 p99 is less than 20% slower than its median.Even at p95, every FaaS platform is further from its own median than EC2 at p99.Source: Figure 1 · p. 1
- Cold starts can increase end-to-end latency, while very slow warm invocations have received less research attention despite their long latency tail.Cold starts have been studied extensively and many mitigation techniques exist.Source: §1 · p. 1
Contributions
- The study characterizes warm-start latency across seven FaaS platforms and an IaaS baseline, finding warm invocations can exceed two orders of magnitude slower than their median in the tail.The baseline is AWS EC2.Source: §1 · p. 2
- Warm calls increasingly contribute to the extreme tail, so the slowest calls are not explained by cold starts alone.The paper introduces warm share to describe the composition of the slowest calls.Source: §1 · p. 2
- The authors screen eleven public datasets and reproduce the observed effects in the four datasets that qualify.They present a reusable method for making heterogeneous serverless traces tail-comparable.Source: §1 · p. 2
2 Background
FaaS delegates infrastructure operations to cloud providers and starts function instances on demand. Requests routed to an available instance are warm starts; requests needing a new instance are cold starts.
FaaS
- FaaS applications use small, event-driven, ideally stateless functions, while providers handle provisioning, scaling, and runtime management.Platforms use virtualization technologies to isolate instances, including runtime isolates, containers, or microVMs.Source: §2 · p. 2
Function Lifecycle
- When no instance is available, the platform creates and initializes one, producing a cold start that can add substantial latency and cost.Initialization can include creating a virtual environment and initializing the language runtime and dependencies.Source: §2 · p. 2
- When an instance is available, the platform routes the request to it as a warm start, and may keep instances alive after requests to increase their likelihood.Platforms terminate instances when demand subsides to save cost and resources.Source: §2 · p. 2
Multi-function workflows
- Complex FaaS applications can comprise multiple functions with explicit dependencies, coordinated through event-based processing and orchestration.The probability of at least one tail event increases with each additional function call.Source: §2 · p. 2
3 Related Work
Prior FaaS research covers benchmarking, cold-start mitigation, and tail latency, while warm-start behavior at extreme percentiles remains comparatively underexplored.
FaaS benchmarking
- Benchmarking research spans realistic workloads and automated microbenchmarks that examine platform behavior, cold starts, latency variability, and cost efficiency.Examples include frameworks by Grambow et al., Kim and Lee, and Yu et al., and microbenchmarks by Copik et al..Source: §3 · p. 2
- Existing benchmarking frameworks often cover few platforms, and modern edge FaaS platforms remain underrepresented in research.Cloudflare Workers (CW), Deno Deploy (DD), and Fastly Edge Compute (FEC) are cited examples.Source: §3 · p. 2
Cold-start mitigation
- Cold-start mitigation research targets either the duration of individual cold starts or their frequency.Strategies include reducing virtualization overhead, merging functions, and predicting invocations to pre-warm instances.Source: §3 · p. 2
Tail latency
- Warm-start tail latency has received little attention, while FaaS tail studies commonly stop at p99 and leave more extreme invocations unexamined.The cited analysis by Ustiugov et al. considers data only up to p99.Source: §3 · pp. 2–3
Adjacent fields
- Work in IaaS and OS performance analysis examines latency beyond p99, including effects of rare high-latency events and virtualization.Dean and Barroso discuss cumulative impact at scale; Xu et al. report virtualization exacerbating long-tail behavior by factors of two to four.Source: §3 · p. 3
4 Analyzing Tail Latency Events
The analysis uses long-running request traces and a virtual-machine baseline to compare warm-start tails across platforms. It examines several percentiles, relative tail severity, and the warm-call composition of the slowest requests.
Dataset and execution
- The primary dataset contains millions of calls to seven commercial FaaS platforms, with recorded start times and cold-start status.Minimal functions and an extended experiment period support tail analysis while reducing user-code effects and temporal sampling bias.Source: §4 · p. 3
IaaS Baseline
- A same-region AWS EC2 t3.nano VM runs the same Node.js HTTP workload under the same load for 11 days as an IaaS consistency reference.The request path avoids a platform-managed control plane or router.Source: §4 · p. 3
Metrics
- The analysis measures Time-To-First-Byte and reports p95, p99, p99.9, and p99.99 to characterize the full range of warm-start tail behavior.Latency percentiles are calculated using warm starts, except for FEC, which reports every invocation as a cold start.Source: §4 · pp. 3–4
- The figure reports tail factors at p95, p99, p99.9, and p99.99, with EC2 showing the lowest overall relative tail latencies and highest consistency.OCF, FEC, Fly.io, CW, and DD show inconsistent tail performance through the day; Lambda and GCF are more stable but have isolated spikes.Source: Figure 2 · p. 4
- The tail factor is a selected latency percentile divided by the median, relating tail performance to each platform's median.It and the latency percentiles are calculated using warm starts, except for FEC, which uses all calls.Source: §4 · p. 4
- Warm share reports the fraction of requests at a latency percentile that are warm starts; above 50% means most of the slowest requests are warm.The metric describes call composition; absolute values compare composition within platforms because cold-request base rates differ.Source: §4 · p. 4
5 Tail Latency Across Seven Platforms
FaaS platforms show heavy, sharply worsening latency tails, while AWS EC2 is substantially more consistent. Warm starts make up much of the extreme tail, and p99.99 latency also varies over time.
Tail Severity Across Seven Platforms
- Up to 947.5× slower than the median, calls in rarely considered percentiles show that warm-start tail latency substantially degrades platform performance.For most platforms, the highest latencies include warm starts, not only cold starts.Source: §5 · p. 4
- Beyond p99, the tail steepens, with the largest single absolute increase in tail factor across every platform and the IaaS baseline occurring from p99.9 to p99.99.For DD, TF rises from 138.46 at p99.9 to 424.05 at p99.99.Source: §5 · p. 4
Warm Share
- At p99.99, more than 50% of calls are warm starts on five of six platforms shown, so the extreme tail is not solely cold starts.The warm share generally falls above p80 before rising again; FEC is excluded because all calls above p80 are cold starts.Source: Figure 3 · p. 5, §5 · p. 6
Comparison to the IaaS Baseline
- FaaS TFp99.99 values range from 6.5 to 947.5, versus 3.4 for AWS EC2; its denser distribution is more consistent.Some FaaS requests are faster than the baseline, but their higher-percentile performance is inferior.Source: Figure 4 · p. 5, §5 · p. 5
Tail Severity Across Seven Platforms
- Ignoring values above p99 improves DD’s distribution mean by up to 33%, while removing Fly.io’s top 0.01% improves its mean by 13%.The reported gains illustrate how a small fraction of extreme values affects the mean.Source: §5 · p. 5
Diurnal Variation at p99.99
- At p99.99, GCF, Fly.io, and Lambda show isolated substantial spikes or dips, while OCF, CW, FEC, and DD show less isolated, longer instability.Baseline spikes are up to two orders of magnitude lower than those of OCF, DD, and Fly.io.Source: §5 · p. 5
Tail Severity Across Seven Platforms
- Absolute maximum latency is a single outlier, unlike p99.99, which aggregates between 28 and 192 calls and is statistically more robust.Treat the maximum as a rare worst-case event, not a stable performance indicator.Source: §5 · p. 6
- Fly.io’s TFp100 is approximately 6149, corresponding to 42 s; DD and CW are roughly 4240 and 712, while GCF is 631.FEC is omitted from this comparison because it has no warm starts.Source: §5 · p. 6
6 Validation with Existing Datasets
Across existing datasets, tail-latency amplification and warm-tail behavior recur across platforms, supporting the conclusion that the effects are not unique to the primary dataset. The validation compares heterogeneous traces after isolating platform performance where possible.
6.1 Methodology
- Validation traces were made tail-comparable by requiring per-request latency and cold/warm labels, then isolating platform overhead from execution time when possible.Joosen et al. provide frontend rather than client RTT, limiting comparability; Huawei25 infers cold starts from the first request on each podID.Source: §6.1 · p. 6
6.2 Results
- Across the analyzed literature, tail-latency amplification and warm-tail behavior recur, making it unlikely that the effects are artifacts of the primary benchmarking framework.Table 2 includes datasets only when per-request latency and cold/warm status are available; Huawei25 infers that status from podID.Source: §6.2 · pp. 6, 9, Table 2 · p. 7
- Lambda20β’s p99.99 latency is 14,623 ms at a 29 ms median, versus 1,703 ms at a 92 ms median for Lambda20α.The paper attributes the difference to experiment setups: relatively constant load in Lambda20α versus deliberate elasticity testing in Lambda20β.Source: §6.2 · pp. 6–7
- Lambda20β’s warm share falls to 35.9% at p99.05, then rises to 87.5% at p99.9; Lambda20α reaches about 98% at p99.95.Overall cold-call rates are 1.63% in Lambda20β and 0.13% in Lambda20α.Source: §6.2 · p. 7, Figure 6 · p. 7
- Azure20’s tail analysis has limited validity because more than 95% of requests failed, and its sample includes only four calls at p99.9.Its reported warm share ranges from 91% to 100% across p80–p100.Source: §6.2 · pp. 7–8
- GCF20 and GCF23 show stronger tail-factor increases than the newer platform in the primary dataset, while their warm-share behavior differs sharply.GCF20 reaches p99.99 latencies of 28,000 ms versus 275 ms for v2; GCF23 has 0% warm calls from p80 to p99.9, amid differing cold-start rates.Source: Figure 5 · p. 6, §6.2 · p. 8
- In Huawei25, the 1,000 mc / 1,024 MB configuration has a p99.99 tail factor of 59,514, while 25,800 mc / 32,768 MB has a tail factor of 5.6.Huawei25 also shows pronounced diurnal variability: its p99.99 spans more than an order of magnitude, and cold starts rise from 0.37% at p95 to 5.6% at p99.95.Source: §6.2 · p. 8
- IBM20’s warm share falls from about 80% at p95 to 47% at p99.7, then rises to 66% at p99.95.IBM20’s p99.99 tail factor of 5.6 corresponds to 19.7 s of platform latency and a 3.51 s median.Source: Figure 6 · p. 7, §6.2 · p. 8
7 Discussion & Future Work
FaaS platforms exhibit warm-start tail latency that cold-start mitigation alone cannot address, and even the most stable analyzed platforms remain less consistent than comparable IaaS. The discussion considers why these tails matter, their scope in real workloads, and possible mitigation.
Warm-Start Tails as a New Problem
- Even when cold starts are excluded, FaaS platforms show heavy latency tails and perform considerably worse than comparable IaaS deployments.The authors report this in the primary dataset and validate it with additional datasets from different years and platforms.Source: §7 · p. 9
Workflow Effect
- As workflows compose more function calls, the probability of at least one tail-latency event increases; under independence, it is 1 − (1 − p)^n.Because independence is not guaranteed in practice, this probability is an upper bound.Source: §7 · p. 9
Warm-Start Tails as a New Problem
- Warm-start tails persist at p99.9 and p99.99, and cold-start mitigation cannot address them.The authors attribute the tails to platform overhead such as control-plane behavior and request routing; diurnal patterns and occurrence with empty functions support that attribution.Source: §7 · p. 9
Warm-Tail Impact Across Platforms
- Even the most stable analyzed FaaS platforms underperform a comparable IaaS deployment; eliminating cold starts would not remove this consistency gap.Lambda, CW, and GCF show relatively stable performance through p99.9, with degradation occasional or generally low at p99.99.Source: §7 · pp. 9–10
Applicability for Low-Latency Workloads
- FaaS prioritizes simplicity, low user-side operational overhead, and scalability rather than promising low-latency execution and request processing.Latency matters for scalability, but the paper says it is not the paradigm’s main focus.Source: §7 · p. 10
Applicability for Real-World Workloads
- Longer real-world function runtimes reduce the relative impact of warm tails and cold starts, although many applications and invocations run for under one second on average.The cited workload characterization reports average execution times below one second for 82% of applications and 96% of invocations.Source: §7 · p. 10
Mitigating Warm-Start Tails
- Client-side retries may avoid long waits for unusually slow requests expected to be warm, but distinguishing warm from cold in-flight requests is difficult and retries do not address the root cause.Retrying a cold start could worsen overall performance.Source: §7 · p. 10
8 Conclusion
Warm-start tail events remain a source of FaaS latency variation as cold-start impact decreases, and extreme percentiles can matter for user-visible workflows. The authors conclude that tight tail guarantees remain difficult to achieve with FaaS and that evaluations should report extreme cold and warm percentiles separately.
Conclusion
- As cold-start impact decreases on modern offerings, warm-start tail events become a comparatively more prominent source of FaaS tail latency.The authors argue that improving perceived latency consistency increasingly requires understanding and mitigating warm-start tails, not only cold starts.Source: §8 · p. 10
- Percentiles beyond p99, particularly p99.9 and p99.99, can capture rare warm slowdowns that dominate the extreme tail and affect end-to-end latency.The authors highlight latency-critical and multi-function workflows as especially relevant contexts.Source: §8 · p. 10
- FaaS evaluations should report p99.9 and p99.99 separately for cold and warm invocations and include the sample count behind those values.Source: §8 · p. 10
- FaaS platforms do not match comparable IaaS latency consistency even when considering only warm invocations.For workloads requiring tight tail guarantees, IaaS-style deployments remain difficult to replace with FaaS, even under favorable conditions.Source: §8 · p. 10
- Similar tail-latency and warm-start effects appear in multiple existing datasets, suggesting they are not artifacts of a single dataset or testing methodology.The authors describe these effects as recurring across platforms and over time.Source: §8 · p. 10