Source-linked AI summary
Pushing CPU Speech Synthesis to the Wall: Extreme Inference Tuning under Serverless Architecture and Billing
Pakorn Nathong, Kunat Pipatanakul
TL;DR
Instance-billed serverless platforms charge for CPU and memory over an instance’s lifetime, including idle periods. This paper reorganizes TTS serving around that bill with bounded per-request parallelism and a reclaimable lifecycle. On Kokoro-82M, the system reduces cost per audio-hour by 4.1× relative to PyTorch defaults.
Problem
Instance-billed serving charges CPU and memory over the container’s lifetime, so idle inference state contributes to cost while scaling to zero makes the next request pay a cold start.
Source: §1 · p. 1Method
The system bounds per-request parallelism and releases inference state and billable page-cache memory after idle periods while retaining the server process and compile cache.
Source: §1 · p. 1, §3.1 · p. 3, §3.2 · p. 3Results
Takeaways & Limitations
Under bursty traffic, request sizing alone can increase total cost, while lifecycle reclamation converts the busy-time efficiency gain into lower serverless cost.
Source: §6.1 · p. 4The study covers one model family, one approach runtime, two CPU architectures, and a synthetic length-mixed trace; the negative optimization results are scoped accordingly.
Source: §6.4 · p. 5, §8 · p. 6
Abstract
from arXiv · showhide
Instance-billed serverless platforms charge for CPU and memory over the lifetime of a warm instance, making idle inference state a direct serving cost. We present billing-aware neural text-to-speech (TTS) serving on serverless CPUs, optimizing CPU-seconds and GB-seconds rather than throughput or latency alone. Conventional runtimes are poorly suited to this setting: per-request parallelism causes CPU contention under concurrency, while warm instances retain gigabytes of billable inference and page-cache state. We address these costs with request-sized concurrent inference, which bounds per-request CPU parallelism, and a reclaimable instance lifecycle, which releases inference state and page-cache memory after idle periods while retaining the server process and compile cache. On Kokoro-82M, our system achieves 2.71 audio-seconds per CPU-second versus 0.89 with ONNX Runtime defaults and reduces cost per audio-hour from $0.0631 with PyTorch to $0.0153, a 4.1x reduction. Idle billed memory falls from 8.7 GB to 1.33 GB, while restoration reaches first audio in 2.2 s versus 7.7 s for a PyTorch cold start. Under bursty traffic, lifecycle reclamation is essential for translating inference efficiency into lower serverless cost.
1 Introduction
Instance-billed serverless TTS makes idle memory part of the serving cost, so optimizing only throughput or latency misses the lifetime CPU and memory bill. The paper addresses busy and idle costs with bounded per-request parallelism and a reclaimable lifecycle, reporting lower Kokoro-82M serving cost.
1 Introduction
- Billing-aware serving targets CPU-seconds and GB-seconds accumulated over an instance’s lifetime, rather than throughput or latency in isolation.The motivating example notes that a PyTorch server retaining 8.7 GB incurs idle cost, while scaling to zero makes the next request wait 7.7 s for first audio.Source: §1 · p. 1
- The design bounds per-request CPU parallelism for concurrent requests and releases inference state and billable page-cache memory after idle periods while retaining the process and compile cache.These mechanisms target the busy and idle components of the bill, respectively.Source: §1 · pp. 1–2
- 2.71 audio-seconds per CPU-second and $0.0153 per audio-hour, 4.1× below PyTorch defaults and 3.1× below ONNX Runtime defaults, on Kokoro-82M.Under bursty traffic, request sizing alone leaves the idle-memory bill, while the reclaimable lifecycle removes that component (Figure 1).Source: §1 · p. 1, Figure 1 · p. 2
2 Motivation and System Observations
The section frames instance-billed serving as a lifetime CPU-and-memory cost problem, then documents contention under concurrent inference and substantial retained memory during idle periods. These observations motivate allocating CPU parallelism across requests and adding an intermediate lifecycle state between warm and process-cold.
2.1 Serverless Billing Changes the Objective
- The memory meter includes cgroup page cache, making retained model files billable until reclaimed; under this billing model, reducing idle memory is a cost optimization.The objective is CPU and memory accumulated over the instance lifetime, not throughput or latency alone.Source: §2.1 · p. 2, §2 · p. 2
2.2 Concurrency and Request Parallelism
- At four clients, ONNX Runtime and OpenVINO defaults achieve only 0.89 and 1.35 audio-seconds per CPU-second, respectively, as their per-request thread pools compete for CPUs.Each runtime sizes its request pool to the machine’s physical cores, although requests cannot be continuously batched.Source: §2.2 · p. 2
2.3 Idle Memory and Instance Lifecycle
- Ten minutes after the last response, resident memory remains 8.7 GB for PyTorch, 4.0 GB for ONNX Runtime, and 5.2 GB for OpenVINO.On bursty traffic, this warm-idle memory dominates the bill for configurations without a release mechanism.Source: §2.3 · p. 2
- A fresh process takes 5.7–7.7 s to reach first audio across evaluated backends, while a warm instance continues incurring idle memory cost.The section motivates a state that relinquishes most billed inference memory but restores faster than a process restart.Source: §2.3 · p. 2, §2.1 · p. 2
3 Design
The design addresses CPU contention and warm-idle memory retention with two complementary mechanisms: bounded per-request parallelism and a reclaimable instance lifecycle. The lifecycle releases inference state after idle periods while retaining a faster restore path than process restart.
3 Design
- The design pairs bounded concurrency for independent requests with a released state between warm-idle and process-cold.Figure 2 depicts the fixed inference pool above and release-and-restore lifecycle below.Source: §3 · p. 2, Figure 2 · p. 2
3.1 Request-Sized Concurrent Inference
- Request-sized concurrent inference bounds per-request parallelism and serves independent requests through a fixed pool of inference streams.The evaluated configuration uses eight streams with two threads each on a 32-vCPU container, with chunked synthesis allowing the first audio chunk to return before the utterance completes.Source: §3.1 · pp. 2–3, Figure 2 · p. 2
3.2 Reclaimable Instance Lifecycle
- After a quiet period, the reclaimable lifecycle frees the compiled model and request-pool state, returns unused allocator memory, and evicts compile-cache files from billable page cache.The server process and on-disk compile cache remain; the next request reconstructs inference state from that cache, with release synchronized against active inference.Source: Figure 2 · p. 2, §3.2 · p. 3
4 Implementation
The implementation uses a common HTTP serving path across three inference backends and makes key backend and lifecycle settings configurable at startup. The OpenVINO backend implements state release and reconstruction from an on-disk compile cache.
4 Implementation
- The common HTTP serving shim applies the reference text-to-phoneme front end, runs inference, and streams audio in chunks for PyTorch eager, ONNX Runtime, and OpenVINO.Measurements cover the complete path from text input to audio output.Source: §4 · p. 3
- Backend, thread and stream counts, compile-cache directory, and idle timeout are configurable at startup.The OpenVINO lifecycle implementation frees inference state, trims allocator memory, evicts compile-cache files from page cache, and reconstructs the compiled model from its on-disk cache.Source: §4 · p. 3
- The implementation uses no custom kernels.The artifact is publicly available.Source: §4 · p. 3
5 Experimental Setup
The experiments evaluate a fixed Kokoro-82M text-to-audio workload across CPU serving configurations, using repeatable, output-equivalent measurements. They compare runtime and lifecycle choices under sustained and bursty demand.
5.1 Model and Workload
- Kokoro-82M’s fp32 community ONNX export was tested on a fixed 16-utterance corpus spanning 13–523 characters and 13–476 tokens per chunk.The shared text-to-phoneme front end makes measurements cover the complete request path from text to 24 kHz audio.Source: §5.1 · p. 3
5.2 Environment and Measurement
- Experiments ran on shared-tenancy Intel and AMD CPU containers, with closed-loop in-container clients and only repeatable, output-equivalent optimizations retained.Primary runs used 32 vCPUs on Intel; transfer runs used 48 vCPUs on AMD. Key configurations were repeated at least twice.Source: §5.2 · p. 3
5.3 Configurations and Metrics
- The complete system used eight two-thread OpenVINO streams, a compile cache, and a 45 s quiet window, compared with five runtime baselines.A synthetic 24-hour trace divided two audio-hours evenly across eight bursts; metrics included efficiency, first-audio latency, idle memory, cold-start and restore latency, and cost.Source: §5.3 · p. 3
6 Results
The results assess end-to-end serving cost, isolate request sizing and lifecycle reclamation, then examine optimization headroom and transfer across architectures and models.
6.1 End-to-End Serving Cost
- 2.71 audio-seconds per CPU-second and $0.0153 per audio-hour are achieved by the complete system, 4.1× below PyTorch defaults and 3.1× below ONNX Runtime defaults.Table 1 measures efficiency and first-audio latency at four concurrent clients.Source: §6.1 · p. 4, Table 1 · p. 4
- Under bursty traffic, request sizing alone leaves 12.1 GB resident and costs $3.24 daily, while adding lifecycle reclamation reduces the complete-system bill to $0.38.Warm-idle memory accounts for 95% of the PyTorch bill in this setting.Source: §6.1 · p. 4
6.2 Request-Sized Concurrent Inference
- A sixteen-thread budget across four OpenVINO streams raises efficiency from 1.35 to 2.31 audio-seconds per CPU-second and lowers four-client median first-audio latency from 1.26 to 0.56 s.Expanding the pool to eight streams raises efficiency further to 2.71 audio-seconds per CPU-second.Source: §6.2 · p. 4, Table 2 · p. 4
- Sized configurations scale to eight clients and then plateau near 38× realtime, with additional requests primarily increasing queueing rather than throughput.CPU use stays between 14.6 and 14.9 of 32 logical CPUs across the plateau.Source: §6.2 · p. 4, Figure 3 · p. 4
6.3 Reclaimable Instance Lifecycle
- After a 45 s quiet period, resident memory drops from 12.1 GB to 1.33 GB; compile caching reduces process cold start from 6.7 to 5.4 s, while release leaves sustained efficiency unchanged.The eight-stream pool raises warm-idle memory to 12.1 GB before release.Source: §6.3 · pp. 4–5, Figure 4 · p. 5
- A request after release reaches first audio in 2.2 s, compared with 6.2 s for a process cold start of the same configuration.The released state retains the process and compile cache while relinquishing inference state.Source: §6.3 · p. 5
6.4 Optimization Headroom
- For the tested graph and hardware, static-shape compilation changes per-call time by -3.3% within repeat-run variability, while padding a 36-token utterance to 128 tokens produces 3.1× as much audio; neither optimization is retained.The profile suggests limited end-to-end headroom from these techniques for this graph, and broader CPU optimization across other models and hardware is left to future work.Source: §6.4 · p. 5
6.5 Transfer
- On Zen 5, the design reaches 3.26 audio-seconds per CPU-second versus 0.69 for PyTorch defaults; separately, pocket-tts retains 1234 MB through ten idle minutes and has no intermediate release state.The backend gap narrows on Zen 5, while pocket-tts is an independently developed model measured in a separate container and not compared directly with Table 1.Source: §6.5 · p. 5, Figure 5 · p. 5
7 Related Work
Related work covers waveform-decoder and streaming approaches, CPU inference stacks, and serverless strategies; this paper instead sizes CPU across concurrent TTS requests and reclaims billable state within a warm container.
7 Related Work
- StyleTTS 2 uses HiFi-GAN- and iSTFTNet-based waveform decoders [6] [7], while streaming systems reduce first-output latency through incremental generation.The paper targets cross-request concurrency and idle state while preserving the Kokoro-82M graph, weights, and output.Source: §7 · p. 5
- PyTorch [11], ONNX Runtime, OpenVINO, and oneDNN provide optimized kernels and controls that this work builds on.The paper sizes CPU across concurrent TTS requests and manages billable state between bursts.Source: §7 · p. 5
- MArk combines batching, autoscaling, and serverless capacity, whereas this work releases model and page-cache state inside a warm TTS container.The warm container retains faster restore than process cold.Source: §7 · pp. 5–6
8 Limitations
The study’s evidence is bounded to one model family, one runtime family for its approach, two CPU architectures, and a synthetic trace; the authors also identify untested optimizations and limit the transfer check’s scope.
8 Limitations
- The evaluation covers StyleTTS2/Kokoro on Kokoro-82M, OpenVINO for the approach, PyTorch and ONNX Runtime baselines, two CPU architectures, and a synthetic length-mixed trace.The negative results are scoped accordingly.Source: §8 · p. 6
- The study did not evaluate quantization, SMT, or core pinning, and static-shape compilation may help graphs dominated by fusable element-wise operators.The evaluated export is not such a graph.Source: §8 · p. 6
- The cross-architecture and model transfer check covers only the billing and lifecycle claims.Source: §8 · p. 6
9 Conclusion
The conclusion frames serverless CPU TTS around lifetime CPU and memory billing, using request-sized inference and reclaimable instance state. The system reduces cost per audio-hour relative to PyTorch defaults, while further gains depend on the model rather than serving-path tuning.
9 Conclusion
- 4.1× lower cost per audio-hour than PyTorch defaults is reported for StyleTTS2-family TTS after restructuring serving around the billed integral.The design combines request-sized concurrent inference with a reclaimable lifecycle that releases billed memory, including page cache, and restores from compile caches in seconds.Source: §9 · p. 6
- On bursty traffic, the conclusion reports substantially greater cost reduction where warm-idle memory dominates the bill.Source: §9 · p. 6
- Beyond this configuration, added concurrency stops converting into throughput and further serving-side tuning measures at or below noise, leaving further improvement determined by the model.Source: §9 · p. 6