PaperlayerView exact revision on arXiv

Source-linked AI summary

Atlas: Optimizing Deployment of Compound AI Workflows on Heterogeneous Clusters

Milos Gravara, Andrija Stanisic, Stefan Nastic

arXiv:2609.04513v1cs.DCcs.AI

TL;DR

Compound AI deployment must choose model variants and hardware placements under SLOs, but upstream errors and information loss make workflow accuracy difficult to estimate from stage-level metrics. Atlas uses MAP to compose adjacent-stage quality transitions and optimize plans; across four workflows, MAP reaches Spearman ρ = 0.947 while reducing profiling cost by 2.6× on the largest measured workflow.

  • Problem

    End-to-end profiling captures stage interactions but scales poorly, while product-based accuracy surrogates treat stages as independent and can misrank plans.

    Source: §I · p. 2, C. Product Based Accuracy Estimation · p. 3

  • Method

    Atlas uses MAP to compose discretized conditional quality transitions between adjacent stages and selects model variants and hardware placements with a MILP under SLOs.

    Source: §I · p. 2, A. Workflow and Quality States · p. 5, B. Local Transition Model · p. 5

  • Results

    Across four evaluated workflows, MAP reaches Spearman ρ = 0.947 and reduces profiling cost by 2.6× on the largest measured workflow.

    Source: §I · p. 2

  • Takeaways & Limitations

    Atlas-Het matches All-RTX4090 accuracy at 42% lower plan cost at the 400 ms SLO by placing the embedder and reranker on cpu-edge and the LLM on A4000.

    Source: E. Plan Selection on a Heterogeneous… · p. 10

    MAP supports sequential, routed, loop, and composed workflows; fan-out aggregation needs additional operators, and stages need discretizable intermediate quality signals.

    Source: §VIII · p. 12

Abstract

from arXiv · show

Compound AI workflows are increasingly used to serve complex AI tasks by coordinating multiple AI models and software components. This approach enables deployment flexibility, as each workflow stage can expose different model variants and resource requirements, but it also expands the deployment choices. A deployment must choose an execution plan that selects AI models for each compound AI workflow stage and places them on a heterogeneous cluster in order to satisfy SLOs. Deployment optimizers therefore need estimates to compare many candidate plans and identify feasible ones. System metrics can often be profiled per stage and composed according to workflow topology, but accuracy cannot, as errors and information loss at upstream stages affect the accuracy of downstream stages. Existing approaches either profile complete configurations end to end, which scales poorly, or use product-based accuracy surrogates that treat stages as independent and can misrank candidate plans. We introduce Atlas, a framework for optimizing compound AI deployments under SLO constraints. Atlas uses MAP, a Markovian Accuracy Predictor, to estimate configuration accuracy from local conditional accuracy transitions between adjacent workflow stages. MAP discretizes intermediate outputs into accuracy buckets and composes transition profiles according to workflow topology, giving the optimizer an accuracy estimate without exhaustive end-to-end profiling. Atlas formulates execution-plan selection as a mixed-integer linear program that maximizes predicted accuracy subject to SLOs. Across four compound AI workflows, MAP achieves Spearman correlation up to 0.947 while reducing profiling cost by up to 2.6x relative to exhaustive end-to-end profiling. Guided by MAP, the Atlas optimizer selects execution plans within 0.03 of oracle accuracy while reducing deployment cost by up to 42% through heterogeneous placement.

I. INTRODUCTION

Compound AI deployment must select stage variants and hardware placements that meet SLOs, but workflow accuracy depends on interactions between stages. Atlas uses MAP to predict accuracy and optimize execution plans without exhaustive end-to-end profiling.

I. INTRODUCTION

  • An execution plan jointly chooses each stage’s model variant, runtime parameters, and hardware placement to maximize accuracy while meeting SLOs.The cited SLO examples include latency, throughput, and cost constraints.Source: §I · p. 1
  • Upstream errors and information loss change downstream accuracy distributions, so workflow accuracy cannot be estimated by composing system metrics in the same way.Existing approaches either profile full configurations at high cost or use product-based surrogates that overlook stage dependencies.Source: §I · pp. 1–2
  • Atlas profiles adjacent-stage quality transitions, composes them with MAP, and uses a MILP to select model variants and hardware placements under SLOs.The framework supports linear, routed, loop, and composed workflow topologies.Source: §I · p. 2
  • MAP reaches Spearman ρ = 0.947 and reduces profiling cost by 2.6× on the largest measured workflow.The reported ranking correlation is across four evaluated workflows.Source: §I · p. 2
  • On heterogeneous clusters, Atlas matches oracle accuracy at up to 42% lower deployment cost than single-tier strategies.On homogeneous clusters, selected plans are within 0.03 of oracle accuracy across evaluated loads and SLOs.Source: §I · p. 2

A. Compound AI Deployment Optimization

  • Adding a router expands the RAG workflow from 72 to 288 configurations, while adding a feedback loop expands it to 1,728.Each additional stage or variant expands the execution-plan search space.Source: A. Compound AI Deployment Optimization · p. 2

B. End-to-end accuracy profiling

Exhaustive profiling captures stage interactions, but its configuration count grows rapidly; PAS composes standalone stage accuracies but ignores dependencies. Atlas profiles adjacent stages and uses the resulting estimates in its deployment pipeline.

B. End-to-end accuracy profiling

  • Exhaustive end-to-end profiling requires thousands of evaluations even for linear RAG and exceeds one million for the composed workflow with routing and a feedback loop.Adding a variant creates combinations with existing variants; adding a stage multiplies the configuration count.Source: B. End-to-end accuracy profiling · p. 3, Figure 2 · p. 3

C. Product Based Accuracy Estimation

  • PAS Spearman correlation falls from 0.295 on the RAG pipeline to 0.068 on the full composed workflow, with zero Top-5 overlap across all three workflows.Table I compares PAS-Naive rankings against measured end-to-end accuracy.Source: Table I · p. 3, C. Product Based Accuracy Estimation · p. 3
  • PAS multiplies standalone stage-variant accuracy scores, making it cheap to compute but unable to capture dependencies between stages.This independence assumption can lead to poor accuracy estimates.Source: C. Product Based Accuracy Estimation · p. 3

III. ATLAS FRAMEWORK OVERVIEW

  • Atlas takes a workflow, candidate variants, calibration data, and SLOs, then produces a plan assigning a variant and hardware placement to each stage.Its framework has profiling, optimization, and execution phases.Source: §III · p. 3, Figure 3 · p. 4
  • The profiling phase measures adjacent-stage quality transitions and per-variant performance by hardware tier; optimization estimates configuration accuracy and selects a plan under SLOs.The workflow executor deploys the selected variants on their assigned hardware tiers.Source: §III · p. 4

A. System Model

Atlas models a workflow as stages connected by data dependencies, then chooses both stage variants and hardware placements to form an execution plan. It estimates serving feasibility from system profiles and optimizes predicted accuracy under system constraints.

A. System Model

  • Atlas chooses a model variant and hardware class for each workflow stage, jointly determining configuration and deployment placement.A workflow configuration assigns variants; an execution plan adds hardware placement.Source: A. System Model · p. 4
  • Stage-variant-hardware profiles provide tail latency, service capacity, and memory footprint; hardware classes also have aggregate memory limits and hourly costs.Atlas uses these profiles to model deployment system cost and constraints.Source: A. System Model · pp. 4–5
  • Predicted task accuracy depends on the workflow configuration, while placement determines serving behavior.Atlas computes plan accuracy from configuration accuracy through MAP.Source: A. System Model · p. 4
  • Latency is evaluated over feasible workflow paths, including branches and repeated stage invocations, while inter-stage communication is omitted under a negligible-overhead assumption.The system model attributes end-to-end latency only to stage executions.Source: A. System Model · p. 4
  • Throughput feasibility conservatively requires every deployed stage to sustain the full workflow rate.Memory feasibility is enforced per hardware class, and the system-level constraints bound the accuracy optimization.Source: A. System Model · p. 5

B. Problem Definition

Atlas selects an execution plan that maximizes predicted task accuracy subject to latency, cost, throughput, and memory feasibility. MAP supplies the accuracy estimate from discretized intermediate quality states composed across workflow stages.

B. Problem Definition

  • Atlas maximizes predicted task accuracy subject to latency, cost, throughput, and memory feasibility constraints.The problem definition supplies workflow and system profiles, hardware capacities and costs, and operator constraints.Source: B. Problem Definition · p. 5

IV. MAP: MARKOVIAN ACCURACY PREDICTOR

  • MAP estimates end-to-end configuration accuracy from local quality profiles, avoiding exhaustive profiling of complete configurations.The Plan Estimator scores candidate configurations before plan selection.Source: §IV · p. 5

A. Workflow and Quality States

  • MAP represents intermediate per-stage signals as discrete quality buckets whose boundaries are computed per stage from calibration data pooled over variants.The bucketed state is assumed to preserve information needed to predict downstream quality and rank configurations, not all semantic properties.Source: A. Workflow and Quality States · p. 5
  • A bucket trajectory is the sequence of quality buckets an input realizes through the workflow, and MAP uses its distribution to predict terminal accuracy.The terminal bucket is mapped to a scalar accuracy value.Source: A. Workflow and Quality States · p. 5

B. Local Transition Model

MAP estimates quality propagation from adjacent-stage conditional transitions, then composes the resulting distributions according to workflow topology. Its operators cover linear pipelines and routed workflows, as well as their composition.

B. Local Transition Model

  • Adjacent-stage transition matrices model the downstream quality-bucket distribution conditioned on the upstream bucket and variant pair.Rows are estimated from calibration samples with smoothing for low-support cells; conditioning locally keeps profiling cost to adjacent stage pairs while capturing direct quality interactions.Source: B. Local Transition Model · p. 5
  • MAP initializes the first stage with a variant-specific bucket distribution estimated from calibration inputs, then uses conditional matrices for subsequent stage pairs.Fig. 4 depicts the local transition model and one possible bucket trajectory.Source: B. Local Transition Model · pp. 5–6, Figure 4 · p. 6

C. Topology Operators

  • Three topology primitives—sequential pipelines, routed workflows, and loops—propagate bucket distributions and map terminal distributions to predicted accuracy; composed workflows chain these primitives.Source: C. Topology Operators · p. 6

1) Linear Pipelines:

  • For a linear pipeline, MAP factorizes the bucket trajectory across stages and predicts accuracy as expected terminal-bucket quality.Terminal-bucket accuracy is averaged from calibration inputs assigned to that bucket.Source: 1) Linear Pipelines · p. 6

2) Routed Workflows:

  • For routed workflows, MAP combines the input routing-state distribution, router branch probabilities, and branch output-quality distributions to predict accuracy.The input state distribution may come from calibration for a standalone router or from the preceding segment in a composed workflow.Source: 2) Routed Workflows · p. 6

3) Feedback Loop:

MAP models a feedback loop as quality-bucket transitions between producer outputs across evaluator-triggered revisions. It supports both fixed-step and adaptive stopping behavior.

3) Feedback Loop:

  • A feedback transition captures the probability that a producer–evaluator pair moves output quality from one bucket to another.The evaluator variant’s transition behavior represents how it drives quality changes across feedback steps.Source: 3) Feedback Loop · p. 6
  • For a fixed feedback budget K, MAP propagates the initial bucket distribution through K steps and predicts accuracy from the resulting quality distribution.Bucket accuracy is averaged over calibration inputs reaching each bucket after K steps.Source: 3) Feedback Loop · p. 6
  • For adaptive stopping, MAP adds an absorbing stop state and estimates both final quality distribution and expected iteration count per input.Atlas uses the expected iteration count to estimate latency and throughput.Source: 3) Feedback Loop · p. 6

D. Workflow Operator Composition

MAP composes workflow segments by passing bucketed quality distributions between their topology-specific operators. A profiled boundary transition maps one segment’s output buckets into the next segment’s input buckets.

D. Workflow Operator Composition

  • Each composed segment maps an input quality-bucket distribution and segment configuration to an output distribution using its topology operator.The operators cover linear, routed, and feedback-loop segments.Source: D. Workflow Operator Composition · p. 7
  • Because adjacent segments use different bucket boundaries, MAP profiles a boundary transition giving the next segment’s entry quality bucket conditional on the preceding segment’s ending bucket.The next segment then applies its own topology operator.Source: D. Workflow Operator Composition · p. 7
  • Passing quality information across segment boundaries preserves cross-segment dependence without conditioning on the full upstream trajectory.Composition profiling cost remains additive across segment boundaries.Source: D. Workflow Operator Composition · p. 7

E. Complexity Analysis

MAP profiles local transitions and predicts over bucket distributions, with profiling costs that grow additively across workflow primitives and boundaries. Atlas uses these predicted configuration accuracies in a MILP for SLO-constrained deployment selection.

1) Profiling cost:

  • Profiling grows additively across workflow primitives and boundaries, rather than with the full end-to-end configuration space.Adding a variant requires O(V · B) new pairwise transitions; inserting a stage requires profiling only its two adjacent stages.Source: 1) Profiling cost · p. 7

2) Prediction cost:

  • A linear segment with S stages requires O(SB^2) prediction work per configuration, and routed and feedback-loop predictions remain polynomial in bucket count and stage count.A linear segment applies one matrix-vector multiplication over B buckets per stage.Source: 2) Prediction cost · p. 7

V. PLAN OPTIMIZER

  • Atlas casts plan selection as a MILP that maximizes precomputed MAP accuracy under linear deployment constraints.A small cost-minimizing tiebreaker favors cheaper placements among configurations with equal predicted accuracy.Source: §V · p. 7
  • The MILP selects one configuration and assigns each stage a model variant and hardware tier, linking configuration choice to deployment indicators.The selector and deployment indicators are binary variables.Source: §V · p. 7
  • Deployment constraints cover path latency, stage throughput, loop iteration load, tier memory capacity, and hourly cost.A stage inside a feedback loop serves each request up to the loop’s iteration bound.Source: §V · p. 7
  • Although the LP relaxation is convex, integrality makes MILP solving NP-hard; MAP-scored configurations can be pruned when they have lower predicted accuracy at equal resource footprint.MAP reduces profiling cost, while the MILP still represents the configuration space through its configuration selector.Source: §V · p. 7

VI. EVALUATION

Atlas is evaluated on four workflows spanning the topology patterns it models, using accuracy prediction and optimizer-decision tests. The experiments use SQuAD answer F1 and bucketed intermediate quality labels.

VI. EVALUATION

  • The evaluation is organized around prediction quality, profiling cost, optimizer plan quality, and heterogeneous placement.The evaluation section introduces the experimental framework and objectives before presenting results.Source: §VI · p. 8

A. Experimental Setup

  • The evaluation tests whether MAP’s local profiles preserve configuration rankings and whether Atlas’s plans compare favorably with baselines on homogeneous and heterogeneous clusters.Atlas is implemented in Python and uses CBC through python-mip for MILP optimization.Source: A. Experimental Setup · p. 8

1) Workflows and Datasets:

  • Four workflows cover linear RAG, routed RAG, RAG with a generator-critic feedback loop, and a composed workflow combining routing and refinement.W2, W3, and W4 extend subsets of W1 configurations.Source: 1) Workflows and Datasets · p. 8
  • All workflows use SQuAD answer F1 as end-to-end accuracy, with intermediate labels describing retrieval, routing, branch, and refinement quality.MAP uses four quality buckets; boundaries are computed from 200 calibration samples per stage, and 100 held-out samples are used for evaluation.Source: 1) Workflows and Datasets · p. 8

2) Baselines:

MAP provides the strongest configuration-ranking signal across four workflows, while its profiling cost is lower than exhaustive profiling at useful ranking quality. The evaluation also compares its predictions and deployment plans with established baselines under specified workloads and cluster settings.

2) Baselines:

  • The baselines include PAS-naive, PAS-fair, and an end-to-end oracle; optimizer plans are also compared with IPA and Loki on a homogeneous cluster.PAS-fair uses topology-aware terms from MAP’s calibration data; the oracle profiles every configuration end to end.Source: 2) Baselines · p. 8

3) Infrastructure:

  • The heterogeneous evaluation uses RTX 4090, RTX A4000, and x86 CPU tiers; reported optimizer metrics include selected-plan accuracy, p99 latency, and hourly cost.The heterogeneous cluster spans three physical nodes.Source: 3) Infrastructure · p. 8, 4) Metrics · p. 8

B. MAP Prediction Quality

  • MAP has the strongest ranking signal across four workflows, with Spearman correlation from ρ = 0.78 on W3 to ρ = 0.947 on W1.Figure 5 plots predicted against measured accuracy; W1 and W2 cluster close to the diagonal, while W3 and W4 show wider spread.Source: Figure 5 · p. 9, B. MAP Prediction Quality · p. 9
  • MAP’s Spearman advantage over PAS-fair grows from 0.39 on W1 to 0.59 on W4; PAS-naive is anti-correlated on W3 at ρ = −0.37.MAP recovers 60–80% of true top-5 configurations, while PAS-naive has zero overlap on W1, W2, and W4.Source: B. MAP Prediction Quality · p. 9

C. Profiling Cost

  • At 1,840 measurements, MAP reaches ρ = 0.93, recovering most reference ranking signal at 2.6× lower cost than exhaustive profiling.Figure 6 compares MAP with PAS baselines and exhaustive end-to-end profiling.Source: Figure 6 · p. 10, C. Profiling Cost · p. 10
  • At |V| = 8, MAP requires 5.3× fewer profiling units, and at |V| = 32 the gap exceeds 80×; at |V| ≤ 4, exhaustive profiling may require fewer measurements.The comparison uses a synthetic scaling scenario with varying variants per stage.Source: C. Profiling Cost · p. 10

D. Plan Selection on a Homogeneous Cluster

Atlas’s plan selection stays close to oracle accuracy on a homogeneous cluster, while heterogeneous placement can reduce cost without sacrificing accuracy. MAP’s ranking quality improves with bucket count, with diminishing gains beyond four buckets.

D. Plan Selection on a Homogeneous Cluster

E. Plan Selection on a Heterogeneous Cluster

F. Bucket Sensitivity Analysis

  • Increasing bucket count to B = 2 yields the largest ranking-quality improvement across workflows, and all workflows plateau by B = 4.W3 dips at B = 3 because equal-frequency quantile boundaries merge failures with mediocre outputs; the evaluation uses B = 4 by default.Source: Figure 10 · p. 11, F. Bucket Sensitivity Analysis · p. 11

VII. RELATED WORK

Prior work addresses inference serving, workflow accuracy estimation, and deployment optimization, but Atlas targets compound workflows where upstream output quality affects downstream accuracy. Its discussion notes limits to MAP’s assumptions and topology coverage, while the conclusion reports plan-selection and cost results across the evaluated workflows.

VII. RELATED WORK

  • The related work spans inference serving, multi-stage workflow accuracy estimation, and constrained optimization for inference deployment.These three areas frame Atlas’s comparisons with prior systems.Source: §VII · p. 11

A. Inference Serving Systems

  • Prior inference-serving systems provide low-latency serving mechanisms and accuracy-aware model selection, but target a single task or model pool at one endpoint.Atlas instead targets compound workflows with downstream accuracy dependent on upstream output quality.Source: A. Inference Serving Systems · pp. 11–12

B. Accuracy Estimation for AI Workflows

  • Product-based accuracy surrogates reuse per-stage measurements but assume independence, while end-to-end profiling captures stage interactions at full-configuration cost.MAP profiles conditional quality transitions between adjacent stages, preserving cross-stage dependencies while scaling with the number of stage pairs.Source: B. Accuracy Estimation for AI Workflows · p. 12

C. Deployment Optimization for AI Workflows

  • Existing deployment optimizers vary in scope: some target homogeneous clusters and limited topologies, while others fix accuracy or optimize heterogeneous placement under accuracy SLOs.Atlas jointly considers variant selection, hardware placement, and topology for compound workflows, using MAP to estimate configuration-level accuracy.Source: C. Deployment Optimization for AI… · p. 12

VIII. DISCUSSION

  • MAP assumes adjacent-stage, first-order quality propagation and that an upstream quality bucket captures information relevant to downstream behavior.Fan-out aggregation requires additional operators, stages without discretizable quality signals may need task-specific instrumentation, and profiling is not eliminated.Source: §VIII · p. 12

IX. CONCLUSION

  • Across four workflows, MAP reaches Spearman correlation up to 0.947 and reduces profiling cost by 2.6×; Atlas selects plans within 0.03 of oracle accuracy, with heterogeneous placement cutting cost by up to 42%.The conclusion reports that MAP’s ranking advantage over product-based approaches grows with workflow complexity.Source: §IX · p. 12

Related papers

Loading 2609.04513v1…