PaperlayerView exact revision on arXiv

Source-linked AI summary

Agar: What-If Analysis for Microservice Applications

Saarth Deshpande, Neeraja J. Yadwadkar, Venkat Arun

arXiv:2610.36636v1cs.DCcs.OScs.PF

TL;DR

Microservice operators need to evaluate changes in workloads, resources, or SLO targets, but deploying each alternative is expensive and composing component predictions is difficult. AGAR composes per-component bounds to answer end-to-end SLO-feasibility questions; across three applications, its autoscaler reduced CPU allocation by up to 36.8% over state-of-the-art horizontal autoscalers while satisfying application SLOs.

  • Problem

    Evaluating counterfactual workload, resource, or SLO changes is expensive, while end-to-end prediction spans interacting components and composing component predictors compounds errors.

    Source: §1 · p. 1

  • Method

    AGAR learns per-component bounds that contain observed behavior and composes them to answer end-to-end SLO-feasibility questions under hypothetical changes.

    Source: §1 · p. 2

  • Results

    Up to 36.8% lower CPU allocation than state-of-the-art horizontal autoscalers while satisfying application SLOs across three microservice applications.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Autoscaling and performance diagnosis can be formulated as counterfactual queries that provide actionable guidance to operators.

    Source: §8 · p. 12

    With substantial measurement noise or outliers, operators may need to introduce slack so a small fraction of data points can violate learned bounds.

    Source: §7 · p. 12

Abstract

from arXiv · show

Microservice operators must reason about how applications would behave under changes that have not yet occurred: different workloads, resource allocations, or SLO targets. Evaluating every alternative experimentally is expensive, but predicting such counterfactuals is difficult because microservices contain many interacting components. End-to-end models must learn behavior over a combinatorial space of conditions, but composing per-component predictors causes prediction errors to accumulate. We address this challenge by changing what each component model must predict. Rather than predicting exact values, we learn simple, interpretable bounds that contain every observed behavior while remaining tight enough to support useful end-to-end conclusions. We build AGAR, a counterfactual performance analysis tool that composes these models to answer end-to-end SLO-feasibility questions. This allows us to formulate autoscaling as a counterfactual query: what is the smallest resource allocation under which an SLO is met? Our evaluation demonstrates that across three microservice applications, AGAR's bounds are actionable: its derived autoscaler reduces CPU allocation by up to 36.8% over state-of-the-art horizontal autoscalers while satisfying application SLOs. Further, AGAR is able to predict operational cost given workload traces, provide guidance on which service's startup time to optimize and localize performance bugs within each application.

1 Introduction

Microservice operators face costly experiments and difficult counterfactual prediction because application behavior spans many interacting components and conditions. AGAR composes per-component bounds to answer end-to-end SLO questions, and its autoscaler lowers CPU allocation while satisfying application SLOs.

1 Introduction

  • End-to-end predictors face a combinatorial space of workloads, resource allocations, and SLO targets, while composing component predictors compounds prediction errors.With 50 component models, naively ensuring 95% confidence in the composition would require each model to be correct roughly 99.9% of the time.Source: §1 · pp. 1–2
  • AGAR replaces exact-value prediction with per-component bounds that contain every observed behavior while remaining precise enough for useful conclusions.An observation outside a predicted range signals that the model must be retrained.Source: §1 · p. 2
  • Simple models absorb unmodeled effects into uncertainty bounds, reducing data needs while remaining interpretable and easier to compose across services.The bounds need to be tight enough to support useful counterfactual conclusions, not to predict behavior exactly.Source: §1 · p. 2
  • Jointly fitting component models to minimize uncertainty in composed predictions is NP-hard, so AGAR uses a greedy algorithm with a log(n)+1 approximation guarantee.The guarantee is for n data points.Source: §1 · p. 2
  • Up to 36.8% lower CPU allocation than state-of-the-art horizontal autoscalers while satisfying application SLOs.AGAR uses its bounds to identify the smallest resource allocation that can meet a target SLO.Source: §1 · p. 2

2 Motivation

Tuning microservice applications requires repeated experimentation, and interactions can make utilization misleading about which service is the bottleneck. AGAR instead reuses component relationships learned from observations and detects when they no longer hold.

2 Motivation

  • HPA requires per-service scaling thresholds and timing policies, and configurations may need retuning as conditions change across applications with many services.Autopilot reports Google services whose changing resource demands required CPU and memory retuning 8–12 times per month.Source: §2 · p. 3
  • Backpressure from a backend bottleneck can raise frontend utilization, making utilization alone misleading about which service should be scaled.Scaling the backend, rather than the frontend, relieves the bottleneck.Source: §2 · p. 3
  • Evaluating startup-time changes, traffic-mix shifts, or changed SLO targets by deployment and retuning turns each question into another experiment.These are counterfactuals about resource cost and SLO satisfaction under conditions not yet tried.Source: §2 · p. 3
  • Erlang requires a non-serving instance to experiment and retrain outside its training parameters, while other systems narrow the search space but still rely on active experiments.Those systems constrain the learning problem to keep it tractable.Source: §2 · p. 3
  • AGAR passively reuses per-service relationships across workloads and flags the first observation that violates them, prompting retraining when reuse is no longer valid.Environment changes exposed previously unseen performance bugs in Online Boutique and TeaStore, invalidating relationships learned on AWS.Source: §2 · p. 3

3 AGAR

AGAR composes per-service models to answer counterfactual SLO-feasibility questions, including how workload, resources, or SLO targets affect required replicas. It uses bounds that fit observed behavior and selects utilization thresholds to support end-to-end decisions.

3 AGAR

  • AGAR relates workload and resource allocation to per-service utilization, then relates utilization to end-to-end SLO satisfaction to answer counterfactual queries.The queries include how resource needs change under different workloads or SLO targets.Source: §3 · p. 4, §3.2 · p. 4

3.1 AGAR Architecture

  • AGAR fits its models from prior endpoint request rates and latencies and per-service CPU and memory measurements, then uses them to answer fixed counterfactual queries.Queries include how changing a service property, such as startup time, affects resources needed to meet an SLO.Source: §3.1 · p. 4

3.2 Models and Modeling Philosophy

  • Utilization bounds represent ranges that hold for all training observations; the utilization model divides workload-dependent utilization across replicas and includes workload-independent utilization and bounded variation.The endpoint-to-utilization relationship is assumed only to be monotonic, allowing different services and endpoints to have different shapes.Source: §3.2 · p. 4
  • The RPS Model predicts future demand from the maximum request rate in the previous t seconds so replicas can be provisioned before demand arrives.The passage reports that future request rate exceeds this prediction in at most 1/E% of observations, where E is the number of endpoints.Source: §3.2 · p. 5

3.3 Composing and jointly optimizing the models

  • For an SLO target, AGAR uses predicted request rates and utilization bounds to determine whether a candidate replica count keeps each service below its SLO-safe utilization threshold.The required replica count follows from keeping the upper utilization bound below the threshold.Source: §3.3 · p. 5
  • Because choosing SLO-safe thresholds is NP-hard, AGAR greedily minimizes additional replica cost per newly excluded violation, with a log(n)+1 approximation guarantee for n training violations.It starts from the largest valid candidate bounds, up to 99%, and selects among models that fit all observed behavior.Source: §3.3 · pp. 5–6

4 Implementation

AGAR separates offline model fitting from online autoscaling, using a per-service control loop and retraining monitors to keep allocations aligned with observed behavior.

4 Implementation

  • AGAR’s Autoscaler evaluates the replica-count equation every 15 seconds and pins each service’s HPA minReplicas and maxReplicas to the computed minimum.It uses current request rates and offline-learned parameters, falling back to HPA if no feasible model exists.Source: §4 · p. 6
  • Utilization-bound violations trigger background retraining, while an endpoint SLO violation triggers a full-application retrain at most once every five minutes.A 60-second per-metric cooldown limits utilization retrains; the SLO monitor checks every 60 seconds.Source: §4 · p. 6
  • Offline fitting and online autoscaling run separately, with serialized asynchronous utilization retrains and coordinated metric snapshots for consistent service decisions.Full retrains use a cross-process lock and do not block the autoscale loops.Source: §4 · p. 6
  • As observations expand coverage of operating conditions, model fits tighten and bound violations become rarer, so replica counts change mainly with genuine demand shifts.Full retraining was required at most once in the reported experiments while utilization bounds were updated during each run.Source: §4 · p. 6

5 Evaluation

Across three applications, AGAR was evaluated against HPA and Erlang on AWS and CloudLab, for autoscaling, diagnosis, counterfactual cost analysis, and training overhead. Results include endpoint-level SLO behavior, environment-specific performance bugs, and cost changes under altered SLOs or startup times.

5.1 Experimental Setup

  • The evaluation used Alibaba-derived workloads across TeaStore, Online Boutique, and Social Network, comparing AGAR with Erlang and hand-tuned HPA on AWS and examining diagnostic cases on CloudLab.Models were fit on the first six hours of 12-hour traces and autoscaler performance was evaluated over the following two hours.Source: §5 · p. 6, §5.1 · p. 7, §5.3 · p. 9

5.2 AGAR as an Autoscaler

  • On Online Boutique, AGAR keeps per-endpoint p95 latency below 50 ms on dedicated hosts and 100 ms on shared hosts, versus a 300 ms SLO; Erlang can miss rare endpoints.Erlang’s overall p95 can remain below the SLO while an endpoint making less than 0.6% of requests reaches median p95 latency of about 1.1 s on shared hosts.Source: §5.2 · pp. 7–8

5.3.1 Online Boutique

  • On CloudLab, correcting Online Boutique’s probe timeout stabilized replica readiness and satisfied the SLO with AGAR’s minimum replica count, whereas scaling alone did not solve the configuration problem.CloudLab storage was roughly 153 IOPS versus 1636 IOPS on AWS, extending startup enough for probes to time out.Source: §5.3.1 · p. 9

5.3.2 TeaStore

  • In TeaStore, an unbounded JVM heap caused unpredictable garbage-collection utilization; bounding heap and metaspace made CPU use predictable and addressed the issue exposed on CloudLab.The issue did not violate the SLO on AWS; slower CloudLab storage amplified GC overhead and queue buildup.Source: §5.3.2 · p. 9

5.3.3 Social Network

  • In Social Network, unbounded Redis memory growth triggered repeated HPA scale-outs that only temporarily reduced per-pod memory; setting a memory cap and eviction policy stabilized usage.The anomaly appeared on both CloudLab and AWS; scaling on CPU instead could end in container memory exhaustion and an OOM kill.Source: §5.3.3 · p. 10, Figure 8 · p. 10

5.4.1 SLO–cost trade-off.

  • For Online Boutique, meeting a 200 ms SLO was predicted to require 36.6% more vCPUs on shared than dedicated t3.2xlarge instances.AGAR swept SLO targets using an unseen workload trace and compared predicted costs with live autoscaler runs.Source: §5.4.1 · p. 10, Figure 9 · p. 10

5.4.2 Varying a Service’s Pod Startup Delay

  • Reducing TeaStore teastore-webui startup time from 30 s to 15 s cut the cost of meeting a 300 ms SLO by 33% on dedicated hosts and 29% on shared hosts.The reductions were from 12 to 8 vCPUs/hr and 14 to 10 vCPUs/hr, respectively; applying the change to other services yielded no cost benefit.Source: §5.4.2 · p. 10

5.5 Training Cost

  • Median utilization-bound retraining latency was at most 14.95 s, just below the autoscaler’s 15 s decision interval; median full-training latency reached 293.23 s at H = 500 and 811.23 s at H = 1000.The training-cost summary spans all applications and environments.Source: §5.5 · p. 11, Table 1 · p. 11

6 Related Work

AGAR differs from prior systems by composing learned per-service bounds to answer end-to-end performance counterfactuals, rather than focusing on other system domains, online allocation, or formal models.

Counterfactual performance modeling and what-if analysis

  • Counterfactual systems for networks, storage, and cloud configurations address unobserved scenarios, but do not compose per-component models in AGAR’s setting.WebPerf predicts latency distributions using request-dependency graphs and profiled cloud API latencies; AGAR uses per-service bounds from application measurements for SLO feasibility.Source: §6 · p. 11

Autoscaling and resource management

  • Most microservice resource-management work targets vertical scaling or online allocation, whereas AGAR targets horizontal autoscaling and also answers feasibility, provisioning, and cost questions.AGAR can identify cases where autoscaling cannot resolve a structural bottleneck.Source: §6 · pp. 11–12

Debugging and root-cause analysis

  • Unlike tracing and resource-remediation approaches, AGAR’s global application model can identify when autoscaling alone cannot reliably satisfy an SLO.Sage identifies a service causing an SLO violation and fixes it by reallocating resources or reducing contention, but focuses on problems addressable by adding resources.Source: §6 · p. 12

Formal modeling and verification

  • Formal verification work often targets correctness or symbolic worst-case behavior from detailed models; AGAR uses application data to reason about end-to-end performance without detailed implementation models.Performal uses upper bounds for worst-case end-to-end latency in single-request distributed systems.Source: §6 · p. 12

7 Discussion and Limitations

AGAR assumes application behavior remains stable over time and recommends retraining after code or hardware changes; slack can accommodate settings with substantial noise or outliers.

7 Discussion and Limitations

  • AGAR assumes fundamental application behavior is stable and recommends retraining after code changes or changes to underlying hardware.These changes mark conditions where previously learned relationships may no longer apply.Source: §7 · p. 12
  • The evaluation needed no slack for utilization or SLO models because the applications were well-behaved, but slack can allow a small fraction of observations to violate bounds when noise or outliers are substantial.The implementation already supports introducing slack with minimal changes.Source: §7 · p. 12
  • Future work includes vertical autoscaling, dynamic resource reallocation, richer models, and automated support for composing models in more complex systems.Richer models require better abstractions and training methods for specifying and learning worst-case behavior.Source: §7 · p. 12

8 Conclusion

AGAR composes telemetry-derived bounded models to assess SLO feasibility under hypothetical changes and frames autoscaling and performance diagnosis as counterfactual queries. The appendix describes a linear-programming approach to fitting utilization models.

8 Conclusion

  • AGAR uses simple models with explicit uncertainty bounds to assess SLO feasibility under hypothetical workload, resource, and SLO changes.The models are built from readily available telemetry.Source: §8 · p. 12

A Utilization model optimization

  • Linear-fractional utilization-model fitting is converted to a linear program using the Charnes–Cooper transformation.An auxiliary variable and scaled variables produce linear constraints for each candidate threshold.Source: A Utilization model optimization · p. 17
  • Training solves two LPs in sequence, minimizing transformed cost first and uncertainty-bound width among equally cost-effective solutions second.The second LP fixes the first optimum while minimizing ˆε_i.Source: A Utilization model optimization · p. 17

B SLO-safe threshold selection

The section describes greedy SLO-safe threshold selection under service-level replica budgets and proves that threshold selection with general monotone costs is NP-hard.

B SLO-safe threshold selection

  • The set-cover construction maps each set to a utilization threshold and each universe element to an SLO-violating observation.Lowering a threshold from 99 to b excludes exactly the observations representing elements in that set; lowering below b cannot improve an optimal solution.Source: B SLO-safe threshold selection · p. 17
  • Threshold selection with general monotone costs is NP-hard, via a polynomial-size reduction from set cover.In the construction, lowering k thresholds to b excludes every violation exactly when the corresponding k sets cover the universe, at cost k additional replicas.Source: B SLO-safe threshold selection · p. 17
  • For each service-level budget, the greedy heuristic lowers thresholds as far as permitted and chooses the smallest additional replica cost per newly excluded violation.It repeats until all violations are excluded; an observation is excluded when utilization in any dimension reaches or exceeds its threshold.Source: B SLO-safe threshold selection · p. 17

C Evaluation Workloads

The evaluation replays Alibaba-derived request-rate traces, using varied demand patterns across application endpoints after training on the preceding six hours.

C Evaluation Workloads

  • The replay traces range from nearly flat demand to bursty fluctuations, sharp spikes, and sustained high load.Different endpoints in the same application may receive qualitatively different demand patterns.Source: C Evaluation Workloads · p. 18
  • Evaluation uses 6–8-hour segments of 12-hour Alibaba-derived request-rate traces, after training on the preceding six hours.One trace is assigned to each application endpoint during replay.Source: C Evaluation Workloads · p. 17, Figure 10 · p. 18

D Evaluating AGAR on CloudLab

CloudLab experiments ran on a Kubernetes cluster, and Figure 11 compares per-endpoint median p95 latency for HPA, Erlang, and AGAR against a 300 ms SLO.

D Evaluating AGAR on CloudLab

  • Experiments ran on a CloudLab cluster of r320 machines running Kubernetes with kubeadm.AGAR's Autoscaler ran on a dedicated node where Kubernetes pod scheduling was disabled.Source: D Evaluating AGAR on CloudLab · p. 18
  • Figure 11 plots per-endpoint median p95 latency for HPA, Erlang, and AGAR, with a dashed line marking the 300 ms SLO.Hollow markers indicate unfixed configurations, and boundary boxes mark average replica counts for each autoscaler configuration.Source: Figure 11 · p. 18

E Effect of Host Interference on AGAR’s Models

Tables 2–4 compare AGAR’s fitted utilization-model parameters on dedicated and shared hosts for a 300 ms p95 latency SLO. Parameter differences indicate shifts in baseline utilization, unexplained variability, and SLO-safe utilization regions.

E Effect of Host Interference on AGAR’s Models

Loading 2610.36636v1…