PaperlayerView exact revision on arXiv

Source-linked AI summary

Gutenberg: Taming Latency-Critical Cloud Services with Near-Data-Processing

Qiushi Lin, Phillip B. Gibbons, Jovan Stojkovic, Yiwei Zhao

arXiv:2609.06691v2cs.DCcs.AR

TL;DR

Latency-critical cloud services put pressure on memory and require low tail latency, while mutable-page replication can make small writes expensive. GUTENBERG combines CPU and NDP execution with fine-grained writable replication and coordinated resource decisions. Across evaluated services, it reduces average and p99 latency by up to 80.4% and 85.8%, while improving isolation and fairness and adapting to workload changes.

  • Problem

    Latency-critical services face memory pressure and strict tail-latency objectives, while page-granular writable replication can make small writes trigger full-page copies.

    Source: §I · p. 1, B. Enabling CPU–NDP Cooperation · p. 5

  • Method

    GUTENBERG combines CPU and NDP execution with a CPU-resident subpage delta buffer and a unified utility that coordinates page actions and request routing.

    Source: §I · p. 2

  • Results

    Across the evaluated suite, GUTENBERG matches or outperforms baselines, reducing average and p99 latency by up to 80.4% and 85.8%, respectively.

    Source: §I · p. 2

  • Takeaways & Limitations

    GUTENBERG’s joint management of computation, data movement, and mutable state is accompanied by improved isolation and fairness and adaptation to changing workloads.

    Source: §I · p. 2, §V · p. 11

Abstract

from arXiv · show

Latency-critical cloud services place growing pressure on memory while requiring isolation, fairness, and predictable QoS. Near-data processing (NDP) reduces data movement by executing requests close to memory, and prior systems further improve locality through caching and replication. However, writes make replica maintenance expensive, while uneven compute and memory traffic can overload a few NDP units and increase tail latency. Existing throughput-oriented schedulers do not fully address these challenges for co-located cloud services. We present Gutenberg, a CPU+NDP for mutable, latency-critical cloud services. Gutenberg stages subpage updates in a CPU-resident delta buffer, allowing hot writable pages to remain replicated without eager full-page synchronization. It also adopts CPU helper cores to assist request execution when NDP execution or replica maintenance becomes costly. An online controller jointly decides page placement, replication, CPU/NDP execution, and routing using access patterns, queue pressure, and feedback from prior decisions. The system further enforces isolation and fair resource allocation across services. We also model-check CPU--NDP coordination protocol for correctness. We evaluate on TailBench using ZSim with Ramulator-calibrated memory timing. Across evaluated services, Gutenberg outperforms prior systems, reducing average and p99 latency by up to 80.4% and 85.8%. It also improves isolation and fairness while adapting to changing workload behaviors.

I. INTRODUCTION

GUTENBERG targets memory pressure and competing QoS demands in multi-service CPU+NDP systems with mutable data. It combines writable replication, CPU assistance, joint scheduling, and resource controls; evaluation reports lower latency alongside isolation and fairness.

  • Multi-service NDP remains comparatively underexplored, while cloud nodes must meet isolation and fairness guarantees amid memory-system pressure.Latency and throughput are often bottlenecked by memory-access latency and limited bandwidth.Source: §I · p. 1
  • Read-only replication excludes mutable pages, compute and data-access loads can diverge under skew, and isolation and fairness can conflict with utilization and locality.Even a small mutable region can prevent replication of hot read-only data elsewhere in its page.Source: §I · pp. 1–2
  • A CPU-resident delta buffer supports replication of mutable pages by staging modified bytes and notifying replicas of dirty regions instead of synchronizing full pages on every write.Accumulated updates are consolidated in bulk after their footprint or update chain exceeds a threshold.Source: §I · p. 2
  • For write-dense pages, CPU helper cores provide an alternative to costly replication, and the CPU–NDP coordination protocol is model-checked.The checked invariants cover single-authority and single-writer safety, legal replica serving, bounded state, and request progress in checked finite configurations.Source: §I · p. 2
  • A unified utility weighs locality, queue relief, and data-movement costs, with page decisions over policy windows and request routing responsive to current queue pressure.Negative feedback updates costs and decision terms using system measurements.Source: §I · p. 2
  • The CPU serves as a trusted control point for service address domains, while per-service budgets and SLO debt help govern shared resources.Services may also use CPU and DPU capacity left idle by co-running services.Source: §I · p. 2
  • Average latency falls by up to 80.4% and p99 latency by up to 85.8% across evaluated services, while GUTENBERG matches or outperforms baselines.The evaluation also reports strong isolation and fairness and adaptation to dynamic workloads.Source: §I · p. 2

II. BACKGROUND AND MOTIVATIONS

GUTENBERG is scoped to multiple cloud services sharing one NDP-enabled server. Its platform places NDP stacks alongside a host CPU, with pages assigned to service domains and home stacks.

A. Target CPU+NDP Platform and Service Model

B. Motivation: Joint Page Management and Execution

Without replication, skewed access patterns can concentrate DDR traffic and queueing even when page footprints are balanced. Hot pages are often requested across stacks but also receive writes, making replication useful and costly; NDP placement decisions must jointly account for computation and data movement.

B. Motivation: Joint Page Management and Execution

  • At 80K QPS, Zipfian access produced a 3.32× maximum-to-minimum DDR-access-volume ratio across stacks, versus 1.12× for distinct-page footprint.At 120K QPS, Zipfian access increased mean DDR-controller queue wait from 56.8 to 72.1 ms, a 27% penalty over uniform access.Source: B. Motivation: Joint Page Management… · p. 3, Figure 2 · p. 4
  • 64.0% of hot-set accesses targeted pages requested from multiple stacks, including 58.2% targeting pages requested from all 16 stacks.Replicas at requester stacks can add local service points and spread accesses across DDR-controller queues.Source: B. Motivation: Joint Page Management… · p. 3, Figure 2 · p. 4
  • Only 15.5% of hot-set accesses targeted read-only pages, while 36.9% targeted pages with write fractions above 40%, making eager writable replication costly.Small writes can trigger page-scale synchronization, motivating sub-page tracking and transfer.Source: B. Motivation: Joint Page Management… · p. 4, Figure 2 · p. 4

C. Related Work: Cloud Services on Other Platforms

  • Unlike CPU, CXL, or distributed-cluster systems, NDP must jointly choose where data reside and where computation executes across a host CPU and lightweight NDP workers.NDP also has limited local memory and more restricted runtime control and workload migration.Source: C. Related Work: Cloud Services on… · p. 4

A. Design Overview

GUTENBERG combines CPU orchestration, CPU helper execution, and a CPU delta buffer to coordinate requests across CPU and NDP workers. Its delta protocol preserves page-level placement while tracking and transferring updates at finer granularity.

A. Design Overview

  • The CPU routes each request to a CPU helper or NDP worker, while separate LLC partitions support helper execution and temporary sub-page update storage.Both worker types can serve requests from the same service.Source: A. Design Overview · p. 4, Figure 3 · p. 5

1) Fine-Grained Writable Replication:

  • GUTENBERG retains 4KB pages for addressing, placement, and replication but divides each page into sixteen 256B regions for replica update tracking.The delta buffer stores and transfers only modified values within a region.Source: 1) Fine-Grained Writable Replication · p. 5, B. Enabling CPU–NDP Cooperation · p. 5
  • For writes, the CPU delta controller merges modified values, versions the changed regions, and sends active replicas compact dirty-region messages rather than the update values.Replicas acknowledge the messages; updated values are fetched from the CPU when a replica later reads a dirty region.Source: 1) Fine-Grained Writable Replication · p. 5
  • A read of a dirty region pulls its buffered updates from the CPU, while reads of clean regions use local replica memory.To reclaim delta entries or change page placement, the controller drains requests, applies updates to active copies, and waits for a full-sync barrier before discarding entries.Source: 1) Fine-Grained Writable Replication · p. 6

2) CPU Helper Execution:

GUTENBERG can route requests for costly pages to CPU helper cores while leaving other pages of the same service on NDP. It also models CPU–NDP coordination to check safety and progress properties across interleavings.

2) CPU Helper Execution:

  • Pages with controller queueing, remote accesses, and frequent replica updates can be temporarily routed to CPU helpers, independently of other pages in the service.The CPU's stronger compute resources and dedicated compute-cache partition support this alternative execution path.Source: 2) CPU Helper Execution · p. 6
  • Switching a page to CPU execution drains routed requests, applies pending deltas, waits for full synchronization, then disables NDP execution and routes future requests to helpers.Returning to NDP waits for helper requests to finish, copies the latest page contents to the home stack and replicas, and re-enables NDP paths after acknowledgments.Source: 2) CPU Helper Execution · p. 6

3) Formally Verified Coordination Model:

  • The TLA+ model checks that CPU and NDP paths are not simultaneously writable, stale replicas cannot complete reads, and transitions preserve request progress.The checks cover interleavings of request service, write publication, replica pulls, delta reclamation, and CPU–NDP switching, including eventual completion under weak fairness.Source: 3) Formally Verified Coordination Model · p. 6

C. Memory-Pattern-Driven Orchestration

GUTENBERG separates periodic page management from per-request worker dispatch, using observed page behavior and queue pressure to choose actions. Its controller predicts latency gains, filters infeasible actions, and admits positive-gain choices subject to resource constraints.

C. Memory-Pattern-Driven Orchestration

  • The paper sketches the orchestrator's functions and parameters, while deferring specific thresholds to future versions.This leaves the threshold settings unspecified in the presented account.Source: C. Memory-Pattern-Driven Orchestration · p. 7

1) Page States and Signals:

  • At each policy window, the controller tracks page accesses, requester stacks, remote accesses, patch density, residency, and worker and DDR-controller queues to select feasible page actions.Actions include migration, replication changes, and enabling or disabling CPU helpers, with choices conditioned on observed access and write patterns.Source: 1) Page States and Signals · p. 7

2) Latency-Oriented Page Actions:

  • For each eligible page action, GUTENBERG predicts amortized per-request latency gain using worker and DDR queueing, execution, memory service, replica visibility, write maintenance, and transition costs.The objective combines these costs rather than separately rewarding locality and queue relief when both result from redirected accesses.Source: 2) Latency-Oriented Page Actions · p. 7
  • The controller ranks positive-gain actions and admits them greedily under stack load, replica and helper capacity, service entitlements, and a per-window mutation bound.It admits at most one action per page and updates projected load before evaluating the next candidate.Source: 2) Latency-Oriented Page Actions · p. 7

3) Per-Request Worker Selection:

The router first derives legal execution paths from a page’s current state, then predicts completion time for each concrete worker. Its estimates distinguish worker queueing, memory-stack contention, and execution and access costs.

3) Per-Request Worker Selection:

  • Page state determines the legal home, replica, and CPU-helper paths available to a request.The router expands those paths into concrete workers and predicts each worker’s completion time.Source: 3) Per-Request Worker Selection · p. 7
  • Worker queueing and memory-stack contention are modeled separately from execution and access costs.The remaining terms distinguish CPU from NDP execution, local from remote memory, and clean from dirty-replica access.Source: 3) Per-Request Worker Selection · p. 7
  • The fast path selects among already legal workers without creating replicas or changing page ownership.The policy window amortizes data movement and changes which workers can be considered.Source: 3) Per-Request Worker Selection · p. 7

4) Outcome-Driven Autotuning:

GUTENBERG tunes action-cost coefficients from observed outcomes rather than fixing them offline. It compares predicted and observed costs after an action’s residence interval, then adjusts bounded coefficients to damp noisy feedback and oscillation.

4) Outcome-Driven Autotuning:

  • Before installing an action, GUTENBERG records predicted queue relief, latency gain, remote traffic, and patch/update cost.After the action’s residence interval, it compares predictions with the observed outcome and updates the corresponding coefficient.Source: 4) Outcome-Driven Autotuning · pp. 7–8
  • Unexpectedly high synchronization cost makes further delta replication less attractive, while greater-than-predicted queue relief raises the value of another local service point.Updates are maintained per action class, with ratios and coefficients bounded against noisy windows.Source: 4) Outcome-Driven Autotuning · p. 8
  • Minimum residence intervals, cooldown, hysteresis, and per-window action bounds damp oscillation in the prediction–action–observation–correction loop.The loop is a runtime autotuner, not an offline system state search.Source: 4) Outcome-Driven Autotuning · p. 8

D. Isolation and Fair QoS

GUTENBERG isolates service memory domains and schedules shared resources through per-service entitlements and SLO debt. Its evaluation compares system designs and workloads under a common simulated hardware configuration.

D. Isolation and Fair QoS

  • Each service’s identity follows requests, delta entries, replicas, and page transitions, allowing the controller to reject cross-service access or updates.Cross-service page sharing is outside GUTENBERG’s scope.Source: D. Isolation and Fair QoS · p. 8
  • Hierarchical elastic scheduling protects each service’s guaranteed share before distributing that share among its users.Entitlements bound candidate workers and page actions; accumulated normalized SLO debt prioritizes services, and fairness is assessed by per-service and worst-service SLO satisfaction.Source: D. Isolation and Fair QoS · p. 8

IV. EVALUATION

  • The evaluation uses ZSim with DDR controllers calibrated against Ramulator and compares methods using the same hardware, arrivals, seeds, and measurement boundaries.Silo and Masstree are primary workloads; other TailBench services broaden coverage across search, translation, speech, image inference, and OLTP.Source: §IV · p. 8, Table II · p. 8, Table III · p. 8

A. Homogeneous-Service Performance

GUTENBERG reduces average and p99 latency in the evaluated services, with gains from queue-aware routing and CPU assistance. Component ablations show that mechanisms contribute differently across workloads and operating points.

A. Homogeneous-Service Performance

B. Component Ablation of Contributions

C. Performance across Different Hardware Profiles

GUTENBERG maintains tail-latency gains across different hardware profiles, workload phases, and co-scheduled services. Its feedback-driven page actions adapt to phase changes, while hierarchical scheduling protects service and user fairness.

C. Performance across Different Hardware Profiles

  • On the high-bandwidth profile, GUTENBERG cuts end-to-end p99 by 53%; on UPMEM-like, GUTENBERG-NDP cuts p99 by 7%, extended to 39% with CPU collaboration.The same controller and page policy are evaluated across profiles with a fourfold change in worker count and more than a 16-fold change in remote bandwidth.Source: C. Performance across Different… · p. 10

D. Adaptation to Workload Changes

  • Transition latency comes within 2.5% of phase-specific steady state by windows 10–14, with A →B reaching 0.99× at window 10 and B →A reaching 1.02× by window 14.The figure plots mean end-to-end latency over 100-request windows, normalized to each phase’s same-seed steady-state mean.Source: D. Adaptation to Workload Changes · p. 10, Figure 9 · p. 10

E. Mixed Services: Isolation and Fair QoS

  • Hierarchical elastic scheduling eliminates all 114,389 starvation windows while keeping both Silo and Masstree below their same-budget standalone p99 latency.It protects each service’s guaranteed share, then allocates that share among users; idle capacity remains borrowable without reducing future guarantees.Source: E. Mixed Services: Isolation and Fair… · pp. 10–11, Figure 10 · p. 11

F. Formal Verification of CPU–NDP Coordination

Across seven services, GUTENBERG reduces average and p99 latency by up to 80.4% and 85.8%. Bounded model-checking runs find no violations of the specified CPU–NDP coordination properties, and additional service results show broad performance coverage.

F. Formal Verification of CPU–NDP Coordination

  • The 14-variable model interleaves request service with replica, delta, migration, and CPU-ownership transitions, while placement changes require a drained page.Routing may choose any legal CPU or NDP target, making correctness independent of the performance policy.Source: F. Formal Verification of CPU–NDP… · p. 11
  • TLC found no specified safety or liveness violations: the safety run reached depth 21 and checked 9 invariants, and the liveness run reached depth 14.The checked properties include unique write authority, legal replica reads, queue draining, and completion of page-state transitions within the bounded policy-mutation budget.Source: F. Formal Verification of CPU–NDP… · p. 11, Table VI · p. 11
  • The verification establishes invariants and bounded progress only in the checked configurations; runtime audits separately check request identity, stale reads, routing, and traffic.Witness properties reached both delta-replica and CPU-owned states.Source: F. Formal Verification of CPU–NDP… · p. 11

G. Coverage Across Additional TailBench Services

  • GUTENBERG-NDP matches or improves on PIMCloud across all five additional services; GUTENBERG lowers p99 across all five, with its largest reduction, 46.9%, on Shore.The figure normalizes end-to-end p99 to the matched PIMCloud run; lower is better.Source: G. Coverage Across Additional… · p. 11, Figure 11 · p. 11
  • Sphinx retains the NDP path and matches GUTENBERG-NDP, while adaptive execution applies CPU help where it contributes and preserves NDP for long speech requests.The passage’s takeaway is that feedback-directed placement remains effective across diverse services.Source: G. Coverage Across Additional… · p. 11
  • On Img-DNN, GUTENBERG-NDP improves SLO goodput over PIMCloud by 12.2%, and GUTENBERG improves it by 16.3%.All methods sustain full fixed-SLO goodput on Xapian, Moses, Sphinx, and Shore.Source: G. Coverage Across Additional… · p. 11

V. CONCLUSION

  • Across seven services, GUTENBERG reduces average and p99 latency by up to 80.4% and 85.8%, respectively.The conclusion attributes the gains to jointly managing computation, data movement, and mutable state.Source: §V · p. 11
Loading 2609.06691v2…