TurboProm: Low Cost and High Performance Metrics Observability at Scale

1 Introduction

• Existing solutions do not simultaneously provide immediate backwards-compatible deployment, improved cost-latency tradeoffs, and minimal impact on analytics fidelity.Other timeseries databases preserve fidelity but offer limited cost-latency benefits and lack backwards compatibility; query-specific designs or coarse sampling sacrifice other requirements.Source: §1 · p. 1
• MO workloads exhibit compact semantic-preserving summaries, query predictability across time, and duplicate computation across queries and time.These observations motivate analyzing workload structure rather than treating queries as isolated requests.Source: §1 · p. 1
• TurboProm estimates query results through semantically equivalent operations on summarized data instead of running the workload directly on raw metrics.For example, summaries can support median and average computations without scanning all raw data.Source: §1 · pp. 1–2
• An offline Planner profiles a deployment and creates a reformulation plan; a streaming Summarizer exports summaries, and a QueryEngine intercepts queries while unsupported queries fall back to the legacy system.The implementation is designed as a backwards-compatible drop-in accelerator for metrics observability deployments.Source: Figure 1 · p. 1, §1 · p. 2
• TurboProm introduces SDSs as atomic summarization units and combines heuristic summary-aware planning, sketch-based streaming summaries, and a QueryEngine that estimates from precomputed SDSs.Queries without an optimized estimation plan use the legacy metrics observability system.Source: §1 · p. 2
• 2x lower query cost and 10x lower query latency over baselines across diverse workloads at small scales; estimated whole-workload benefits at hyper-scale range from 1.2X–900X.For isolated query types, latency reduction reaches up to 3800X.Source: §1 · p. 2

2 Background on metrics observability (MO)

• Metrics observability instruments, collects, stores, and analyzes numerical telemetry from software deployments; Prometheus is its de-facto standard.Operators use the telemetry for alerting, anomaly detection, autoscaling, and CI/CD tasks.Source: §2 · p. 2

2.1 Motivating use-case scenarios

• Performance monitoring ranks services by tail latency, tracks sustained resource saturation, and identifies excessive error rates; anomaly detection compares distributional measures across subpopulations or with historical behavior.Trend analysis also examines resource consumption and capacity pressure over longer windows, with queries often rerun every few hours.Source: §2.1 · pp. 2–3
• MO workloads combine continuously repeated queries into ensembles of spatio-temporal aggregations over time-series data.Queries filter and aggregate across timestamps and metadata labels, and can combine results with binary arithmetic.Source: §2 · p. 2, §2.1 · p. 3

2.2 Problem: Cost, scale, and latency

• Operators seek low total cost of ownership and low query latency, but both rise as cloud deployments scale.The section describes total cost as including provisioned-machine costs for ingesting metrics and running queries.Source: §2.2 · p. 3, Figure 2 · p. 3
• At 640k metrics/sec, the evaluated Prometheus workload reached 40 seconds p95 query latency and 14 cores p95 CPU usage.The setup ran 10 PromQL queries each second on a 16-core machine while varying offered load.Source: §2.2 · p. 3

3.1 High-level idea

• Synopsis-based views let TurboProm reuse computation across queries, such as estimating median and p90 CPU usage from one KLL sketch.The views also support estimates from either approximate synopses or exact representations.Source: §3.1 · pp. 3–4
• For a 5-minute query repeated every 30 seconds, disjoint 30-second synopses avoid reprocessing each data point in 10 query windows.The appropriate chunk synopses are merged to estimate each sliding-window result.Source: §3.1 · p. 4
• Computing synopses as data streams in moves some work from query time to ingest time, reducing query latency.Query estimation can then occur when queries are issued.Source: §3.1 · p. 4

3.2 Challenges

• Existing time-series, table, and RDD abstractions do not fit materialized synopses, which connect queries to metrics data and require a new abstraction.That abstraction must bridge diverse query types and synopsis algorithms while remaining extensible.Source: §3.2 · p. 4
• Planning must choose views and rewrites across intractable query workloads, while the data plane must continuously materialize synopses and integrate with existing deployments.Synopsis materialization as a first-class principle conflicts with existing systems’ abstractions, motivating a design from scratch.Source: §3.2 · pp. 4–5

3.3 System overview

• TurboProm’s Self-Describing Synopses (SDSs) provide an abstraction for materialized synopses, including atomic summarization units over metrics streams.A window average for a specified label filter and time window is one example of an SDS.Source: §3.3 · p. 5
• The Planner takes a high-level SQL or PromQL workload and produces a plan specifying which SDSs to materialize and how to rewrite queries.Users may optionally provide accuracy requirements for queries.Source: §3.3 · p. 5
• The Summarizer, StorageEngine, and QueryEngine materialize, store, and use SDSs; queries that cannot be estimated fall back to the existing backend.The fallback gives every query a backend option, including complex, ad-hoc, or non-aggregation queries.Source: §3.3 · p. 5

4.1 Core abstraction: Self-Describing Synopsis

• Each Self-Describing Synopsis records its data scope, synopsis algorithm and configuration, and serialized payload, enabling query compatibility checks without raw data and extension to new algorithms.The scope includes the metric, time range, optional metadata filters, and partitioning labels.Source: §4.1 · pp. 5–6

4.2 TurboProm’s control plane: Planner

• The Planner addresses complex repeating-query workloads and an intractable plan space by decomposing queries, generating feasible candidates, and optimizing their selection using offline benchmarks and an analytical cost model.It aims to minimize ingestion and query cost while respecting accuracy and latency requirements.Source: §4.2 · pp. 6–7

4.2.1 Decomposing RQEs into RAQEs

• RAQEs contain one aggregation without nested aggregations or binary operators; this is treated as safe because their results are composable and aggregations dominate computational cost, but unsupported RAQEs fall back to the existing backend.TurboProm assumes innermost aggregations usually account for most cost and supports only RQEs whose constituent RAQEs can be satisfied.Source: §4.2.1 · p. 7, §4.2.2 · p. 7

4.2.2 Generating feasible plans

• Candidate plans vary by synopsis mergeability, invertibility, and subpopulation awareness: for 3.2M label combinations, non-aware KLL requires 3.2M SDSs, while a suitable aware synopsis could use one.Mergeable summaries can trade lower ingest cost for query-time merging; invertible summaries can support sliding-window updates by subtracting the oldest window.Source: §4.2.2 · pp. 7–8

4.2.3 Formulating the optimization problem

• The optimization selects one candidate plan per RAQE to minimize ingestion plus frequency-weighted query cost, enforcing accuracy and latency constraints while charging shared SynopsisPlans only once.Empirical operation benchmarks and analytical models estimate costs, latency, and accuracy; partial sharing between distinct SynopsisPlans is not modeled and may overestimate ingestion cost.Source: §4.2.3 · p. 9

4.2.4 Putting everything together

• The Planner decomposes each RQE into RAQEs, generates SynopsisPlan–EstimationPlan candidates, then solves the optimization and maps each RAQE to a deployed plan.Source: §4.2.4 · p. 9

4.3 TurboProm’s data plane

• The data plane incrementally summarizes incoming metrics through extensible dataflow DAGs that share filters and groupings, while the StorageEngine buffers recent SDSs in memory and asynchronously persists older partitions.Queries access in-memory summaries first and retrieve persisted summaries when their time range requires them; the data plane also includes a QueryEngine for estimating results from SDSs.Source: §4.3 · pp. 9–10
• The QueryEngine maps parsed queries to EstimationPlans that select SDSs and operators, tracks prior results for subtraction, and forwards unsupported queries to the existing backend.Its synopsis-agnostic core shares a synopsis library with the Summarizer, enabling extension to new synopses.Source: §4.3 · p. 10

5 Implementation

• The Planner parses PromQL queries into ASTs and outputs ingestion and estimation configurations as YAML.It is implemented in 3k lines of Python and uses py-promql-parser.Source: §5 · p. 10
• The Summarizer turns ingestion configuration into Arroyo streaming pipelines and sends Self-Describing Synopses to the QueryEngine over Kafka.It uses Prometheus remote_write to receive metrics.Source: §5 · p. 10
• The QueryEngine estimates results from several sketch and exact-accumulator types and forwards unsupported queries through Prometheus’ HTTP API.Its library includes KLL, CountMinSketch, Hydra-on-KLL, and exact accumulators for rate, increase, min, max, sum, and count.Source: §5 · p. 10

6.1 Setup

• The evaluation uses Alibaba, Google Borg, and synthetic metrics, and measures querying cost as p99 CPU usage while excluding ingestion cost.Prometheus and VictoriaMetrics are tested out of the box; reported experiments use two CloudLab nodes.Source: §6.1 · pp. 10–11, §6 · p. 10

6.2 End-to-end validation

• Across 10 random workloads at 8k measurements/sec, TurboProm provides both median-latency and p99-querying-cost benefits over Prometheus.The authors report similar benefits against VictoriaMetrics, though those results are not shown.Source: §6.2 · p. 11, Figure 10 · p. 11

6.3 What-if Scenarios

• OneS queries provide the most latency benefit among the profiled AtomicQuery types, with each output key aggregating 25K points versus 1K for OneT.The four profiles are OneS quantile, OneS sum, OneT quantile, and OneSOneT distributive sum.Source: §6.3 · p. 11, Figure 11 · p. 11
• TurboProm’s latency benefit over Prometheus increases with query lookback window and data cardinality.The scenarios vary cardinality for OneS quantile queries and both lookback windows and cardinality for OneT quantile_over_time queries.Source: §6.3 · p. 12, Figure 13 · p. 12
• 1.2X to 900X overall latency speedup is estimated across workload mixes and data shapes for hyperscale deployments.The estimates cover 10 workload mixes and four data shapes; the authors report consistent benefits across real and synthetic mixes.Source: Figure 12 · p. 11, §6.3 · p. 12

7 Other Related Work

• Proprietary hyperscaler observability systems often retain the classic data-and-query model, which TurboProm’s ideas may also benefit.The authors describe these systems as proprietary architectures and data stores.Source: §7 · p. 12
• Horizontal scaling is complementary to this work, which focuses on query processing on a single node.Applying TurboProm’s ideas to scale-out deployments, distributed querying, and long-term storage is future work.Source: §7 · p. 12
• Logs and tracing also face cost and scaling bottlenecks, but whether TurboProm’s insights generalize to them remains future work.Prior efforts address compact logging and reducing tracing costs.Source: §7 · p. 12
• Database approximate-query-processing work does not directly target metrics-observability workloads or TurboProm’s deployment constraints.The authors also note a lack of concerted query planning for summary-based computation.Source: §7 · p. 12

8 Discussion

• Future work includes expanding SketchOpLib, improving the heuristic and sketch parameters, and testing horizontally scaled deployments.Hosted Prometheus services often charge by ingestion, so customers may not directly benefit, though vendors could pass on cost benefits; the authors also point to possible use beyond metrics observability.Source: §8 · p. 13

• Monitoring system health becomes a significant bottleneck and cost sink as systems scale in size and complexity.Source: §9 · p. 13
• A first-principles analysis motivates recasting the problem using a semantic-preserving summary view.Source: §9 · p. 13
• TurboProm applies this idea in a practical drop-in accelerator that yields significant savings.Source: §9 · p. 13
• The semantic-preserving summary view and named-data summary abstraction may apply to other observability and data-processing systems at scale.Source: §9 · p. 13