PaperlayerView exact revision on arXiv

Source-linked AI summary

Managing Iterative Hybrid Quantum-Classical Optimization as a First-Class Scientific Workflow

Giuliana Siddi Moreau, Maria Laura Clemente, Lorenzo Pisani, Manuela Profir, Marco Pinna, Marco Moro, Lidia Leoni

arXiv:2609.30532v1cs.DC

TL;DR

The application-level loop must manage submissions, failures, and records that explain a run, while generic workflow systems lack several loop-specific behaviors. The paper models these needs in a workflow layer; across two real decomposition patterns, orchestration shares were 78.4% and 6.2%, respectively.

  • Problem

    The application-level loop must decide what to submit after a round, how to handle failed submissions, and what to record for later explanation, needs not fully met by generic workflow systems.

    Source: §I · p. 1

  • Method

    The workflow layer adds feedback-driven termination, subproblem recovery, within-round backend failover, and backend-agnostic provenance to a dynamic task graph.

    Source: §I · p. 2, A. Dynamic Graph with Feedback-Driven… · p. 4

  • Results

    Iterative consensus devoted 78.4% of time to orchestration versus 6.2% for hierarchical partitioning, characterizing distinct workloads rather than ranking them.

    Source: A. Orchestration Cost (RQ1) · p. 6

  • Takeaways & Limitations

    For iterative workloads, the number of device crossings per round is a resource-management knob, determined by the block solver’s execution mode rather than solver quality.

    Source: §VIII · p. 8

    The paper does not establish which solver or backend produces better solutions; that is a separate benchmarking question.

    Source: §VIII · p. 8

Abstract

from arXiv · show

Today's Quantum Processing Units (QPUs) are too small and too noisy to solve large combinatorial optimization problems directly, so practical hybrid solvers split a problem into pieces and iterate a decompose-solve-aggregate loop over whatever backends are available: classical heuristics, simulators, emulators, or a QPU. In practice, the loop is a driver script. It sits on top of the quantum-HPC middleware, handling task generation, provenance, recovery, and portability. Instead, we treat the loop as a scientific workflow and ask what a workflow layer adds to a generic workflow management system and QPU-sharing middleware. Two decomposition patterns from real applications, iterative consensus (ADMM) and hierarchical partitioning, turn out to stress the orchestration layer very differently: over 120 managed runs, orchestration took 78.4% of end-to-end time for the iterative pattern, almost all of it in a per-round barrier, but only 6.2% for the hierarchical one. Our workflow model adds four things a generic engine does not have: a termination predicate residing in the task graph that reads the previous round's residuals, subproblem-level recovery with warm-start and quorum-deferred aggregation, failover from a QPU to a classical replica within a round, and a provenance schema that describes CPUs, simulators and QPUs with the same fields, including the shot budget included. We report the cost of each layer on our engine, show that speculative re-execution enable runs to complete under injected failures that stall an unmanaged driver. We also use the same provenance to give per-device latency tails across simulators, emulators and IQM QPUs.

I. INTRODUCTION

Hybrid optimization loops must manage submissions, failures, and records that explain a run, while driver-script orchestration offers little explicit graph structure, provenance, or automated recovery. The paper treats the loop as a workflow and adds loop-specific orchestration capabilities across decomposition patterns.

The orchestration problem

  • The application-level loop must decide what to submit after a round, how to respond to failed submissions, and what to record for later explanation.Source: §I · p. 1

Hybrid optimization context

  • Realistic binary quadratic models are divided into subproblems, solved on backends, and combined over rounds; the paper focuses on running this loop rather than on a particular decomposition algorithm.Source: §I · p. 1

The orchestration problem

  • Driver scripts typically submit subproblems, wait, and stitch answers, with little explicit task-graph structure, provenance, or automated retries.The scripts are also tied to one site’s scheduler, while QPUs are scarce, remotely queued, subject to calibration drift, and sometimes fail.Source: §I · p. 1

Contributions

  • The workflow model adds feedback-driven termination, warm-start and quorum-deferred recovery, within-round backend failover, and backend-agnostic provenance that includes shot budgets.The paper also characterizes two real decomposition patterns and evaluates orchestration cost, injected-failure behavior, and device latency tails.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

Generic workflow systems and quantum–HPC middleware provide task enactment, provenance, retries, and resource scheduling, but do not supply all loop-specific orchestration behavior. The paper positions its layer above these systems and treats several hybrid application patterns as workloads.

Workflow terminology

  • A workflow has dependent tasks, and it is dynamic when its task set becomes known only during enactment; this loop is dynamic through convergence and adaptive subproblem generation.Source: §II · p. 2

Generic workflow systems

  • Runtime graph construction in Parsl, Dask, and PyCOMPSs supports dynamic fan-out, while Pegasus and Covalent provide task-level provenance and retries.Source: §II · p. 2
  • Generic workflow systems do not provide retries from a previous iterate, quorum-based aggregation, within-round QPU-to-classical failover, or shot-budget-aware provenance.Source: §II · p. 2

Quantum–HPC middleware

  • Quantum–HPC middleware decides where and when circuits run, while this layer decides what to submit, whether a round is done, and how to respond to failure.The recorded per-device latency tails are described as input a QPU-sharing policy would want.Source: §II · p. 2

The application class

  • Decompose–solve–aggregate methods are the workload here, and related patterns also occur in quantum chemistry, circuit cutting and knitting, and domain-decomposed simulation.Source: §II · p. 2

III. HYBRID OPTIMIZATION AS A WORKFLOW

The paper describes hybrid optimization independently of a particular solver, then identifies a shared decompose–solve–aggregate loop and two structurally different decomposition patterns.

A. The Decompose–Solve–Aggregate Loop

  • Each iteration decomposes a residual problem into state-dependent subproblems, dispatches them independently to interchangeable backends, and aggregates results to update the global assignment and test convergence.The loop ends when improvement stops or a budget is exhausted.Source: A. The Decompose–Solve–Aggregate Loop · p. 3

B. Decomposition Patterns and Fan-out

  • ADMM repeatedly fans out coupled block BQMs and performs consensus and dual updates before residual testing, whereas hierarchical partitioning stitches clustered routing solutions and re-solves infeasible branches.The two patterns place different orchestration demands; nested partitioning with ADMM is expressible but was not evaluated.Source: B. Decomposition Patterns and Fan-out · p. 3

C. Why a Static Task Graph Does Not Fit

The workflow requirements arise from data-dependent fan-out, heterogeneous stochastic backends, and the need to handle late or failed samples while preserving replayable state. The proposed model grows a task graph iteratively and uses aggregation results to determine whether it continues.

C. Why a Static Task Graph Does Not Fit

  • Because fan-out depends on current data and backend results can be partial, noisy, late, or failed, a fixed task graph does not capture the loop’s execution needs.Intermediate state must also be recorded well enough to replay a run.Source: C. Why a Static Task Graph Does Not Fit · p. 3

D. Requirements

  • The stated requirements are feedback-aware dynamic task generation, per-subproblem provenance, retryable tasks with backend failover, and portability across workflow systems and schedulers.Source: D. Requirements · pp. 3–4

IV. Workflow Model and Orchestrator

  • The orchestrator owns dynamic-graph semantics, provenance, fault tolerance, and portability, while the solver and task-enactment engine remain replaceable.Source: §IV · p. 4

A. Dynamic Graph with Feedback-Driven Termination

  • The graph emits subproblem tasks from each iteration, then aggregation updates global state and emits either another iteration or a terminal node; a termination node reads aggregation’s residuals.Decomposition and aggregation/termination operators let iterative, hierarchical, and nested patterns share one model.Source: A. Dynamic Graph with Feedback-Driven… · p. 4

B. Provenance Schema for Hybrid Runs

The provenance schema records subproblems, iterations, and runs with backend-independent fields. Fault handling combines subproblem recovery with whole-run rescheduling, while an executor interface supports portability across execution environments.

B. Provenance Schema for Hybrid Runs

  • Each subproblem, iteration, and run writes a corresponding record, with the same fields for CPU, simulator, and QPU backends.The executor fills in the device descriptor and shot budget at solve time.Source: B. Provenance Schema for Hybrid Runs · p. 4

C. Fault Tolerance and Straggler Mitigation

  • Aggregation proceeds after a quorum of subproblems returns, and invalid blocks warm-start from their previous-iteration values.Tasks are idempotent; timeouts, retries, and speculative re-execution can cover slow or rejected QPU jobs with classical replicas.Source: C. Fault Tolerance and Straggler… · p. 4
  • Invalid runs are rescheduled with bounded retries and exponential backoff, resuming from a per-iteration solver-state checkpoint.The controller can optionally switch to a more reliable backend, and the reschedule count is recorded.Source: C. Fault Tolerance and Straggler… · pp. 4–5

D. Portability Layer

  • The same loop maps through an executor interface to Slurm, Grid Engine, or a Python task engine without changing decomposition, aggregation, provenance, or fault-tolerance code.A remote-QPU executor adds embedding, queue handling, and shot budgeting.Source: D. Portability Layer · p. 5

V. HPC AS A QUANTUM-PREPARATION TESTBED

The paper uses HPC to build and test workflow capabilities for hybrid quantum-classical optimization before fault-tolerant QPUs are widely available. Its evaluation measures orchestration rather than solver quality, using two real-application workloads and controlled failure and straggler conditions.

V. HPC AS A QUANTUM-PREPARATION TESTBED

  • HPC can exercise the workflow's fan-out, dependencies, aggregation contention, and injected QPU-like failures and latencies at scales unavailable to current QPUs.Only subproblem solving is backend-tied; classical samplers can stand in during development.Source: §V · p. 5
  • The readiness levels progress from dynamic feedback-driven graphs to replayable provenance, resilience, and portable quantum-aware execution.L1–L3 can be measured on HPC; L4 is exercised partly on IQM QPUs and otherwise with simulators, emulators, and a queue model, and the levels are not a validated metric.Source: §V · p. 5

VI. EVALUATION METHODOLOGY

  • The evaluation isolates workflow orchestration rather than solution quality, holding solvers and instances fixed.Orchestration time is client-side end-to-end time minus backend solve times recorded in BlockRecords.Source: §VI · pp. 5–6
  • Runs use HPC clusters and include VQE, QAOA, and DCQO samplers with simulators, noise-aware emulators, and cloud-accessed IQM QPUs.The platform includes a 124-node HPC cluster and a Slurm-managed virtual-machine cluster.Source: §VI · pp. 5–6
  • The workloads pair hierarchical partitioning for fleet-routing variants with ADMM iterative consensus for renewable-energy-community allocation and energy sharing.The evaluation asks about orchestration cost, resilience under injected failures and stragglers, and per-device latency tails and shot cost.Source: §VI · p. 6
  • Each configuration runs 10 times, with medians and interquartile ranges reported and p50 and p99 denoting median and 99th-percentile tail.Controlled queue-model failure and straggler rates approximate a remote QPU; orchestration time includes transport, queue and barrier wait, serialization, and workflow bookkeeping.Source: §VI · p. 6

A. Orchestration Cost (RQ1)

Orchestration costs differ sharply between the two workload patterns: iterative consensus incurs repeated barrier waits, while hierarchical partitioning pays mostly for stitching. The reported figures characterize the managed engine with provenance enabled.

A. Orchestration Cost (RQ1)

  • 61.8% of the 783.18 s median end-to-end time across 120 managed runs was orchestration, or 484.08 s.The reported orchestration time includes transport, queue and barrier wait, serialization, and graph and provenance bookkeeping.Source: A. Orchestration Cost (RQ1) · p. 6
  • 78.4% of iterative-consensus time and 6.2% of hierarchical-pattern time was orchestration, characterizing patterns run on different services and instances rather than ranking them.Iterative cost is almost entirely recurring per-round barrier wait; hierarchical cost is mostly stitch wait, with bounded re-dispatch for infeasible clusters.Source: A. Orchestration Cost (RQ1) · p. 6, Table V · p. 7
  • The iterative pattern's orchestration share grows with iteration count, while the hierarchical pattern pays mostly once for fan-out and stitching.The paper identifies reducing device round trips as a scheduler opportunity for the iterative pattern.Source: A. Orchestration Cost (RQ1) · p. 6
  • The measurements are for the managed engine with provenance enabled; runs without provenance or on an unmanaged driver were not yet measured.The authors state that comparison was underway.Source: A. Orchestration Cost (RQ1) · p. 7

B. Feedback-Driven Termination (Provenance View)

The workflow uses recorded ADMM residuals to decide whether to create another fan-out or stop. The recorded trajectory makes the stopping point checkable after the run.

B. Feedback-Driven Termination (Provenance View)

C. Resilience Under Injected Failures (RQ2)

Injected-failure tests show speculative re-execution can sustain completion without shifting first-attempt latency, while production losses were absorbed before rescheduling. Per-subproblem backend provenance also supports device-level latency and shot-budget reporting.

C. Resilience Under Injected Failures (RQ2)

  • In production, quorum-deferred aggregation and warm-starting absorbed all losses before rescheduling, despite 4736 failed or timed-out blocks.No block was re-dispatched and no speculative copy was launched in production.Source: C. Resilience Under Injected Failures… · p. 7
  • As injected failures rise, speculative re-execution keeps runs completing where an unmanaged driver stalls, without moving the first-attempt distribution.The first-attempt distribution sets typical iteration makespan; re-dispatched work appears in a separate tail.Source: C. Resilience Under Injected Failures… · p. 7, Figure 4 · p. 7

D. Backend Portability and Real-QPU Reach (RQ3)

  • Each BlockRecord records the resolved provider, device type, device name, and shot budget for its subproblem.This backend descriptor remains useful when failover sends different subproblems in one iteration to different samplers.Source: D. Backend Portability and Real-QPU… · p. 7
  • Of eight backends used, three were QPUs, including IQM Emerald and Garnet at 8192 shots per circuit.Table VI reports per-device blocks run, latency tails, and shot budgets.Source: D. Backend Portability and Real-QPU… · p. 7

VIII. DISCUSSION AND LIMITATIONS

The discussion links recurring orchestration costs to resource-management choices and bounds the paper's claims about solution quality and evaluation scope. The conclusion argues for treating iterative hybrid decomposition as a scientific workflow and reports resilience under injected failures.

VIII. DISCUSSION AND LIMITATIONS

  • For iterative workloads, submission frequency is the resource-management lever: solver execution mode fixes how often each round crosses to the device.The discussion says this choice, rather than solver quality, is the knob for a scheduler above QPU-sharing middleware such as.Source: §VIII · p. 8
  • The paper does not establish which solver or backend gives better solutions; that comparison is left to benchmarking.Source: §VIII · p. 8
  • Most orchestration evaluation is HPC rehearsal with an injection model, which does not reproduce real queue dynamics at scale.The orchestration cost is reported only for the managed engine; nested patterns, multi-scheduler portability, and fan-out scalability remain unevaluated.Source: §VIII · p. 8

IX. CONCLUSION

  • The two decomposition patterns differ by an order of magnitude in orchestration demands, with iterative consensus dominated by a recurring per-round barrier.The conclusion identifies that barrier as the dominant iterative-pattern cost.Source: §IX · p. 8
  • The paper argues that HPC is the place to build and test iterative hybrid quantum-classical workflows before fault-tolerant QPUs arrive.Its workflow layer adds feedback-driven termination, recovery, within-round backend failover, and device-and-shot-budget provenance.Source: §IX · p. 8
  • The conclusion lists unmanaged-driver and generic-WMS baselines, nested-pattern evaluation, more real-QPU execution, and cross-facility enactment as next steps.Source: §IX · p. 8
Loading 2609.30532v1…