PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantum Reinforcement Learning for Cost and Delay Tradeoffs in Quantum Cloud Orchestration

An N. H. Phan, Dang Van Huynh, Muhammad Usman, Hoa T. Nguyen

arXiv:2609.27446v1cs.LGcs.AIcs.DCcs.ETquant-ph

TL;DR

Heterogeneous quantum resources under uniform time-based pricing make orchestration a tradeoff among execution cost, delay, and fidelity. The paper proposes QRLQ, which integrates PQCs into a D3QN to jointly optimize execution and waiting time. Simulations report lower cost and delay than heuristic baselines, fidelity within 2% of a fidelity-greedy policy, and comparable performance to classical DRL with fewer trainable parameters.

  • Problem

    Heterogeneous QPUs and uniform time-based pricing make task placement a tradeoff among execution cost, queue delay, and fidelity.

    Source: §I · p. 1

  • Method

    QRLQ integrates parameterised quantum circuits into a duelling double deep Q-network and models orchestration as a sequential decision problem that jointly minimizes execution and waiting time.

    Source: §I · p. 2

  • Results

    QRLQ achieves 5–11% lower mean cost than availability-based and rotation-based heuristics, 17% and 82% lower mean delay than the strongest and weakest heuristic baselines, respectively, and fidelity within 2% of a fidelity-greedy policy.

    Source: §I · p. 2, §VI · p. 12

  • Takeaways & Limitations

    The simulations suggest QRLQ can provide comparable scheduling performance to classical DRL with approximately 72% fewer trainable parameters, supporting the feasibility of parameter-efficient quantum-classical orchestration.

    Source: §VI · p. 12

    The current framework assumes a single quantum task per QPU.

    Source: §VI · p. 12

Abstract

from arXiv · show

Quantum cloud computing, delivered through the quantum-as-a-service (QaaS) model, provides access to quantum computing resources. However, applying uniform time-based pricing across fundamentally heterogeneous quantum resources significantly complicates task orchestration, particularly when addressing the tradeoff between execution costs and system performance. While heuristic methods rely on predefined scheduling rules, classical deep reinforcement learning (DRL) models may require more trainable parameters in this setting. Motivated by the potential of parameterised quantum circuits (PQCs) as compact function approximators, we propose QRLQ, a cost-delay-aware quantum cloud scheduling framework integrating PQCs with a dueling double deep Q-network (D3QN) to dynamically account for both cost and delay. Our simulation results show that QRLQ achieves lower mean cost and delay than the heuristic baselines, achieving a 5-11% lower mean cost relative to availability-based and rotation-based heuristics and reducing mean delay by 17% and 82% relative to the strongest and weakest heuristic baselines, respectively, while retaining execution fidelity within 2% of a fidelity-greedy policy. Compared with the classical DRL baseline, QRLQ achieves comparable scheduling performance while using 72% fewer trainable parameters. This work explores the feasibility of using QRL for task orchestration in quantum cloud environments and demonstrates its potential for cost-delay-aware quantum resource management.

I. INTRODUCTION

Heterogeneous QPUs and uniform time-based pricing create a scheduling tradeoff among execution cost, queue delay, and fidelity. QRLQ applies quantum reinforcement learning to jointly manage cost and delay, and its simulations report better heuristic-baseline cost and delay with fidelity near a fidelity-greedy policy.

I. INTRODUCTION

  • Different QPU specifications affect execution fidelity, delay, and cost, making quantum task placement a multi-objective scheduling problem.Under uniform time-based pricing, slower hardware can increase execution cost, while concentrating tasks on high-performance devices can increase queues and delay.Source: §I · p. 1
  • Across the evaluated IBM backends, shorter execution times generally coincided with relatively high fidelity, while longer execution times did not consistently yield higher fidelity.The evaluation used GHZ, QAOA, and EfficientSU2 circuit families at varying depths; the reported pattern suggests faster QPUs may reduce time and cost without reducing fidelity.Source: §I · pp. 1–2, Figure 1 · p. 2
  • The paper identifies a gap in applying QRL to quantum-cloud orchestration under uniform time-based pricing and proposes QRLQ, a quantum duelling double deep Q-network framework.It formulates orchestration as a stochastic sequential decision problem and describes a hybrid quantum-classical network with PQCs for scheduling across heterogeneous QPUs.Source: §I · p. 2
  • The paper’s stated scope includes exploring QRL feasibility for quantum-cloud orchestration, with future fault-tolerant and error-corrected environments as a target context.Its introduction also reports promising results under evaluated parameter perturbation levels.Source: §I · p. 2
  • QRLQ reduces mean cost by 5–11% relative to availability-based and rotation-based heuristics, and mean delay by 17% and 82% relative to the strongest and weakest heuristic baselines, respectively.Execution fidelity remains within 2% of a fidelity-greedy policy; compared with classical DRL, QRLQ uses approximately 72% fewer trainable parameters with comparable scheduling performance.Source: §I · p. 2

II. RELATED WORK

The related work spans static heuristics and learning-based orchestration, while the proposed system assigns each task using circuit features and dynamic node states. The task model treats requests as independent abstract circuits with execution-relevant characteristics.

A. Heuristic-based Orchestration Approaches

  • Heuristic and mathematical-programming approaches address cost, latency, fidelity, or uncertainty, but can struggle to adapt to changing workloads and queue states.Examples include stochastic programming for total cost and waiting-time penalties, and multi-criteria planning to balance cost and fidelity.Source: A. Heuristic-based Orchestration… · p. 3

B. Learning-based Orchestration Approaches

  • Existing DRL approaches adapt scheduling to system states but mainly target fidelity and completion time; QRL uses PQCs as compact function approximators and remains underexplored for quantum-cloud orchestration.The cited approaches include Rainbow DQNs, PPO, graph convolutional networks, and calibration-informed scheduling.Source: B. Learning-based Orchestration… · p. 3

A. System Architecture

  • A centralized QRLQ broker extracts task features, evaluates them with dynamic QNode states, assigns the task to the node with the highest estimated action value, and then prepares it for execution.The system targets joint minimization of execution time and waiting time because slower nodes can raise cost, while concentrating work on high-performance nodes can increase queues.Source: A. System Architecture · p. 3

1) Quantum Task Model:

  • Each QTask is modeled as an independent abstract circuit workload, with characteristics including arrival time, circuit width and depth, gate counts, final measurements, and required shots.The model assumes a single independent abstract circuit per task.Source: 1) Quantum Task Model · p. 4

2) Quantum Resource Model:

The resource model represents each QNode as a single dedicated QPU with hardware properties and an execution queue. Circuit transpilation to the target topology yields a DAG used to estimate execution time and fidelity.

2) Quantum Resource Model:

  • Each modeled QNode abstracts one dedicated QPU, provides exclusive access to one circuit at a time, and executes tasks non-preemptively.The study focuses on the single-QPU case, although modern hardware may support multiple QPUs and multiprogramming.Source: 2) Quantum Resource Model · p. 4
  • QNode characteristics include available physical qubits, supported native gates, expected availability time, operation-time mappings, and corresponding error probabilities.The hardware varies in qubit capacity and connectivity.Source: 2) Quantum Resource Model · p. 4
  • Mapping an abstract circuit to a target hardware topology produces a transpiled-circuit DAG, whose critical path supports execution-time estimation and whose operation errors support fidelity evaluation.The time estimate uses hardware operation times, while fidelity evaluation uses error probabilities.Source: 2) Quantum Resource Model · p. 4, Figure 3 · p. 4

C. Orchestration Performance Model

The performance model links execution cost to QPU runtime and combines execution time, waiting time, and fidelity measures to characterize scheduling outcomes.

C. Orchestration Performance Model

  • Under homogeneous pricing, execution cost is directly proportional to QPU execution time.The unit price per unit time scales the cost of an accepted task assignment.Source: C. Orchestration Performance Model · p. 4
  • Execution time is estimated from the transpiled circuit’s critical path and requested shots, assuming serial shot execution without reset or readout overheads.The model is deterministic across heterogeneous backends.Source: C. Orchestration Performance Model · p. 4
  • A complexity bonus raises the score for higher-complexity tasks, while execution time remains the main guide toward cost-efficient allocation.The bonus is intended to reduce the disadvantage that short, simple circuits may have under execution-time-led scheduling.Source: C. Orchestration Performance Model · p. 5
  • Waiting time runs from task arrival until execution starts, and the model also defines an overall task-delay measure.Waiting time is normalized to form a waiting-time performance score.Source: C. Orchestration Performance Model · p. 5
  • Execution fidelity is approximated by accumulating the success probabilities of physical operations in the transpiled circuit.For each operation, success probability is represented as one minus its physical error probability.Source: C. Orchestration Performance Model · p. 5

D. Problem Formulation

The problem formulation treats orchestration as sequential assignment of arriving tasks to heterogeneous QNodes, with a score balancing execution and waiting time.

D. Problem Formulation

  • The policy assigns each arriving quantum task to a selected node in a cluster of heterogeneous QNodes.The objective is to maximize cumulative scheduling performance over the online sequence.Source: D. Problem Formulation · p. 5
  • The assignment score weights execution-time and waiting-time performance using β, fixed at 0.5 for equal priority.β ranges from 0 to 1 to adjust the tradeoff between the two objectives.Source: D. Problem Formulation · p. 5
  • A valid assignment requires enough physical qubits and compatibility with the node’s native operations and valid qubit operands.Source: D. Problem Formulation · p. 5

E. Markov Decision Process (MDP) Formulation

QRLQ models scheduling as an MDP in which task and hardware state inform node allocation, and rewards guide execution-time and waiting-time optimization.

E. Markov Decision Process (MDP) Formulation

  • At each task arrival, the agent observes the state, chooses an action, receives a scalar reward, and transitions to the next state.The transition advances to the subsequent task and updates the selected node’s ready time.Source: E. Markov Decision Process (MDP)… · p. 5
  • The MDP state combines incoming-task features with features of available QNodes, including task complexity and quantum load, node readiness, and performance characteristics.Features are scaled to [0, 1] and mapped to rotation angles in [0, π] before PQC encoding.Source: E. Markov Decision Process (MDP)… · p. 6
  • Each discrete action selects one available QNode for the incoming task.The action space maps directly to the set of available nodes.Source: E. Markov Decision Process (MDP)… · p. 6
  • The immediate reward combines execution-time and waiting-time performance scores, weighted by the tradeoff parameter β.Source: E. Markov Decision Process (MDP)… · p. 6

IV. THE PROPOSED QRLQ FRAMEWORK

QRLQ combines a PQC feature extractor with duelling double-DQN value estimation for task allocation, using trainable circuit operations to represent resource features and correlations.

A. Overall QRLQ Architecture

  • QRLQ combines Double DQN and Duelling DQN with a PQC replacing the classical dense feature extractor, reducing trainable parameters from 758 to 214 while maintaining classical scheduling performance.The value and advantage streams estimate state value and action-specific importance before producing action values for node selection.Source: A. Overall QRLQ Architecture · p. 6

B. Parameterised Quantum Circuit (PQC) Design

C. Value Observation and Duelling Output

QRLQ encodes quantum-cloud state features with a parameterised quantum circuit and combines quantum readouts with classical duelling value and advantage streams. Its training uses replay-based Double DQN updates and a soft target network to stabilise learning.

C. Value Observation and Duelling Output

  • State features are encoded through rotation gates and entangling CZ gates, then Pauli-Z measurements produce features for classical value and advantage streams.The streams are used to compute action values for feasible hardware-allocation actions.Source: C. Value Observation and Duelling Output · p. 7
  • The duelling output separates state value from action advantage, with the action space corresponding to available QNodes.The model outputs an action value for each feasible allocation action.Source: C. Value Observation and Duelling Output · p. 7

D. Learning Stabilisation and Training Strategy

  • Replay-sampled transitions train online and target networks: the online network selects next-state actions, while the target network evaluates them.Gradient clipping bounds the gradient norm at 1.0, and soft target updates help stabilise Q-value estimates.Source: D. Learning Stabilisation and Training… · pp. 7–8

A. Simulation Setup

The evaluation uses a heterogeneous simulated quantum cloud with workloads drawn from MQT Bench and IBM-device configurations. It compares heuristic, classical DRL, and quantum DRL policies under shared core reinforcement-learning settings.

A. Simulation Setup

  • The simulation uses QSimPy, MQT Bench tasks with 3–30 circuit layers and 2–27 measurement qubits, and five simulated IBM backends with 27 or 127 qubits.Poisson-distributed arrivals model stochastic submissions; idealised simulation isolates algorithmic feasibility from hardware-induced errors.Source: A. Simulation Setup · pp. 8–9
  • Heuristic baselines include earliest availability, fastest duration, round robin, and random allocation policies.The policies are Greedy Earliest Availability, Greedy Fastest Duration, Cyclic Round Robin, and Random Uniform Selection.Source: A. Simulation Setup · p. 9
  • The learning-based baselines are classical Duelling Double DQN and Quantum DQN, which directly benchmarks the proposed hybrid architecture.Source: A. Simulation Setup · p. 9
  • The evaluated models share core reinforcement-learning hyperparameters, while QDQN additionally scales quantum expectation values toward the target Q-value range.Source: A. Simulation Setup · p. 9

B. Performance Study

Across the evaluated policies, QRLQ combines low mean cost with the lowest mean delay, while keeping execution times competitive and waiting times below those of the heuristic baselines. During training, it reaches the highest mean reward among the compared reinforcement-learning methods.

1) QRLQ Training Performance and Efficiency:

  • QRLQ achieved the highest mean reward, stabilising at approximately 0.775 during the final 100 training episodes with narrower confidence intervals than the baselines.All three reinforcement-learning methods improved rapidly during the first 200 episodes.Source: 1) QRLQ Training Performance and… · p. 9

2) Execution and Waiting Time Performance:

  • QRLQ's execution time beat every heuristic except GFD and was 1.46% lower than QDQN and 2.70% lower than D3QN.GFD had the lowest execution time, but its device concentration could lengthen subsequent queues.Source: 2) Execution and Waiting Time… · p. 10
  • QRLQ maintained waiting times below 35 seconds, outperforming all heuristic baselines and achieving lower average waiting time than the learning-based baselines.D3QN and QDQN also avoided heavy congestion, while QRLQ had smaller variance.Source: 2) Execution and Waiting Time… · p. 10

3) Cost–Delay Tradeoff Analysis:

4) Execution Fidelity Performance Analysis:

QRLQ achieved lower mean cost and delay than heuristic baselines while maintaining execution fidelity within 2% of a fidelity-greedy policy. The analysis also reports its fidelity, computational efficiency, behavior under parameter perturbations, and current simulation scope.

4) Execution Fidelity Performance Analysis:

5) Computational Complexity and Efficiency Analysis:

  • Hardware execution scales as O(l ×(a+b+n)+Pc), while classical simulation scales as O(l × ((a + b) × n + n) × 2n + Pc).Hardware space complexity is O(l × a × n + Pc), compared with O(2n + Pc) for classical simulation.Source: 5) Computational Complexity and… · p. 11

6) Impact of Noise Model and Discussion:

  • As parameter perturbations increased, QRLQ’s waiting time varied less than QDQN’s; both models’ execution times stayed within the classical-baseline range, although QRLQ’s rose gradually.The evaluation perturbed trained PQC rotation angles with Gaussian noise; Fig. 8 compares waiting and execution times across noise levels.Source: Figure 8 · p. 11, 6) Impact of Noise Model and Discussion · p. 11
  • The simulations used five QNodes and an 8-qubit shallow-depth PQC; the appropriate circuit size for larger action spaces remains to be evaluated.Source: 6) Impact of Noise Model and Discussion · p. 11

VI. CONCLUSION AND FUTURE WORK

  • In the reported scheduling comparison, QRLQ reduced mean delay by 17% and 82% relative to the strongest and weakest heuristic baselines, respectively, while maintaining execution fidelity within 2% of a fidelity-greedy policy.Source: §VI · p. 12

5) Computational Complexity and Efficiency Analysis:

  • QRLQ used approximately 72% fewer trainable parameters than classical DRL; its 214 total parameters compare with 758 for classical D3QN.QRLQ required approximately 10.5 hours of training on CPU and GPU, compared with 11.25 hours for QDQN on both and 10.5 hours for D3QN on GPU.Source: 5) Computational Complexity and… · p. 11, §VI · p. 12

VI. CONCLUSION AND FUTURE WORK

  • Future work includes quantum multiprogramming, larger action spaces, distributed quantum execution, iterative hybrid workloads, and validation on physical quantum hardware clusters.Source: §VI · p. 12
Loading 2609.27446v1…