PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Solutions for Hybrid Quantum-HPC Resource Allocation

Roberto Rocco, Simone Rizzo, Matteo Barbieri, Gabriella Bettonte, Elisabetta Boella, Fulvio Ganz, Sergio Iserte, Antonio J. Peña, Petter Sandås, Alberto Scionti, Olivier Terzo, Chiara Vercellino, Giacomo Vitali, Paolo Viviani, Jonathan Frassineti, Sara Marzella, Daniele Ottaviani, Iacopo Colonnelli, Daniele Gregori

arXiv:2508.04217v1quant-phcs.DC

TL;DR

Hybrid HPC-QC workloads face resource-allocation challenges because their classical and quantum phases need different resources at different times. This paper evaluates workflow-based management and malleability, finding that both reduce resource use and malleability improves time-to-solution compared with workflow management. The experiments show dynamic allocation strategies outperform a statically allocated baseline in the tested use case.

  • Problem

    Hybrid applications’ changing classical and quantum resource needs make allocation difficult, especially when quantum resources are scarce and static reservations leave resources idle.

    Source: §III · p. 3

  • Method

    The paper evaluates workflow-based resource management and malleability, which dynamically adjusts resource allocation during hybrid HPC-QC jobs.

    Source: §I · pp. 1–2

  • Results

    Both approaches significantly reduce resource use, and malleability achieves shorter time-to-solution than workflow management, with the temporal gain growing as cluster load increases.

    Source: §I · p. 2

  • Takeaways & Limitations

    In the tested hybrid HPC-QC use case, dynamic resource-allocation strategies show advantages over a traditional statically allocated baseline.

    Source: §V · p. 6

    The authors do not claim a one-size-fits-all solution; the strategies are demonstrated through a representative HPC-QC application.

    Source: §V · p. 6

Abstract

from arXiv · show

The integration of quantum computers within classical High-Performance Computing (HPC) infrastructures is receiving increasing attention, with the former expected to serve as accelerators for specific computational tasks. However, combining HPC and quantum computers presents significant technical challenges, including resource allocation. This paper presents a novel malleability-based approach, alongside a workflow-based strategy, to optimize resource utilization in hybrid HPC-quantum workloads. With both these approaches, we can release classical resources when computations are offloaded to the quantum computer and reallocate them once quantum processing is complete. Our experiments with a hybrid HPC-quantum use case show the benefits of dynamic allocation, highlighting the potential of those solutions.

I. INTRODUCTION

Hybrid HPC-QC systems pair quantum accelerators with classical computing, but scarce quantum resources make efficient allocation a challenge. This paper evaluates workflow-based management and introduces malleability for hybrid jobs, finding both reduce resource use and malleability improves time-to-solution over workflow management.

I. INTRODUCTION

  • Malleability dynamically adjusts a job’s resource allocation, allowing it to release classical HPC resources during quantum computation and reclaim them afterward.This targets under-utilization when only one or two QPUs are available per cluster.Source: §I · pp. 1–2
  • The paper explores malleability for hybrid HPC-QC jobs and a workflow-based strategy, whose use for optimizing resource use in these environments remains uncommon.The workflow implementation is based on StreamFlow.Source: §I · pp. 1–2
  • The evaluation parallelizes a clustering-aggregation application and compares time-to-solution and resource use across three management scenarios on an ad-hoc cluster.The scenarios are no hybrid resource management, StreamFlow workflow management, and malleability; the cluster has three classical compute nodes and a classical quantum emulator.Source: §I · p. 2
  • Both workflow-based management and malleability significantly reduce resource use, while malleability also shortens time-to-solution versus workflow management, especially as cluster load grows.The paper evaluates the approaches against a setting without hybrid resource management.Source: §I · p. 2

II. TOWARDS HPC-QC: CLUSTERING AGGREGATION

The paper adapts clustering aggregation to HPC-QC: parallel classical clustering outputs are combined into a graph, then a quantum-suitable optimization phase finds a consensus clustering. The application iterates until its silhouette score meets a threshold or reaches a loop limit.

II. TOWARDS HPC-QC: CLUSTERING AGGREGATION

  • The application runs k-means, DBSCAN, and hierarchical clustering on separate HPC nodes, then combines their outputs into a graph for clustering aggregation.The application loops until a user-defined silhouette threshold or maximum iteration count is reached.Source: §II · p. 2, Figure 1 · p. 3
  • A valid clustering corresponds to an independent set in the graph, and a maximum independent set yields a consensus clustering intended to mitigate the individual algorithms’ limitations.The aggregation is formulated as a QUBO minimization problem.Source: §II · p. 2
  • The QUBO objective uses cluster weights and a penalty factor λ to discourage selecting overlapping clusters, with binary variables x_i ∈ {0, 1}.The weights are proportional to cluster dimensions, while the penalty discourages overlap.Source: §II · pp. 2–3
  • The quantum phase receives the weighted adjacency matrix, solves the QUBO with Simulated Annealing, and the application stops at silhouette score greater than 0.8 or after 10 iterations.The best result so far is chosen if the iteration limit is reached.Source: §II · p. 3

III. TOOLS FOR HYBRID RESOURCE ALLOCATION

Hybrid HPC-QC applications shift between classical and quantum phases, but static resource allocation can hold resources idle. The section motivates workflow orchestration and malleability as approaches to that mismatch.

III. TOOLS FOR HYBRID RESOURCE ALLOCATION

  • Reserving both resource types for an entire job is inefficient when only one type is used at a time.The section notes this is particularly consequential when quantum resources are scarce.Source: §III · p. 3
  • Reserving classical resources while waiting for a quantum task to be scheduled also leaves those resources unused.The section presents workflow execution and malleability as approaches to improve allocation efficiency.Source: §III · p. 3
  • Hybrid applications use classical and quantum resources at different times, while their changing requirements conflict with static allocation.The mismatch is a source of resource inefficiency.Source: §III · p. 3

A. Workflow

The workflow approach models an application as dependent tasks placed across execution locations. StreamFlow represents the clustering use case as an iterative workflow spanning classical HPC nodes and a quantum location.

A. Workflow

  • A hybrid workflow is represented by a workflow graph, available execution locations, and a mapping from steps to locations; quantum-capable steps can be assigned to quantum locations.At runtime, the workflow system can choose a classical or quantum implementation depending on quantum-resource availability.Source: A. Workflow · p. 3
  • StreamFlow extends CWL with deployment-location topology and iterative-workflow support, and uses pluggable connectors for execution environments.Its connectors can be extended to integrate quantum-device APIs or queues.Source: A. Workflow · pp. 3–4
  • The clustering application is modeled as an iterative three-step workflow: compute clusterings, aggregate them through QUBO, and evaluate the silhouette score.The example maps clustering to three classical locations, QUBO aggregation to one quantum location, and score evaluation to a classical location.Source: A. Workflow · p. 4

B. Malleability

Malleability lets an application resize its classical allocation at runtime. DMR coordinates resource and process changes, while the hybrid code releases most classical resources during quantum processing and resumes from a checkpoint.

B. Malleability

  • Malleability lets parallel applications change their resource allocation, including node or process counts, during execution.The allocation can grow or shrink as system conditions change.Source: B. Malleability · p. 4
  • DMR coordinates reconfiguration among the resource management system, parallel distributed runtime, and performance monitor.It provides an API for incorporating malleability into HPC applications.Source: B. Malleability · p. 4
  • At synchronization points, DMR can reconfigure resources and process counts, then resume execution with the new job shape.The application supplies a checkpoint and restart function to preserve state across reconfiguration.Source: B. Malleability · p. 4
  • After the classical stage, the hybrid code keeps only the root MPI process active during quantum processing, relinquishing the remaining resources.A checkpoint stores the loop iteration and whether execution can proceed to the quantum stage; the restart function restores that state.Source: B. Malleability · p. 4

IV. EXPERIMENTAL CAMPAIGN

The experiments use a dedicated SLURM cluster with separate compute and quantum-emulator partitions. The quantum phase is simulated with a classical QUBO solver, with artificial delays used to approximate longer quantum jobs.

Experimental setup

  • The test cluster uses SLURM 23.02.7, with three CPU-only compute nodes and a separate quantum-emulator node.The compute nodes each have two AMD EPYC 7543 CPUs and 256 GB of memory; the emulator node has two AMD EPYC 7282 CPUs and 512 GB.Source: §IV · p. 4

Towards quantum

  • The quantum phase is formulated as QUBO and solved with simulated annealing on a classical node; artificial delays approximate quantum-job durations.The authors state that replacing the placeholder node with a quantum device would require no changes to their configurations or methodologies, while device execution times differ by technology.Source: §IV · p. 5

A. Experimental Results

Dynamic approaches trade wall time against classical node-seconds, with results shaped by contention and quantum-job duration. In concurrent two-minute-job runs, workflow and malleability finish earlier and use fewer resources than baseline; the no-delay timing result is explicitly algorithm-dependent.

A. Experimental Results

  • Without resource contention, workflow uses the fewest node-seconds but has the poorest wall time, while malleability is a compromise; both conserve resources with negligible time-to-solution impact.These comparisons average five runs with two-minute quantum jobs.Source: A. Experimental Results · p. 5, Table I · p. 6
  • With concurrent workloads and two-minute quantum jobs, workflow and malleability finish earlier and use fewer resources than baseline.Malleability keeps at least one MPI process active during the quantum phase, enabling immediate resumption when quantum processing ends even if all originally requested resources are unavailable.Source: A. Experimental Results · p. 5, Figure 2 · p. 6
  • Without artificial quantum delays, malleability completes both simulations in 648.61 s versus 549.60 s for a single execution, but the authors attribute this outcome to algorithm-specific classical-phase timings.The authors caution that this result is not a general trend and expect limited resource savings from either dynamic approach for short quantum jobs without contention.Source: A. Experimental Results · pp. 5–6

B. Discussion

For long-running quantum phases, workflows favor resource efficiency, while malleability may suit time-critical work, especially in clusters with long queue times. Neither approach is a drop-in replacement for static-resource applications.

B. Discussion

  • Workflows use resources very efficiently, while malleability may be more appropriate when time to solution matters, especially with long queue times.The experiments suggest both approaches suit hybrid workloads with long-running quantum phases; current metrics exclude queue times.Source: B. Discussion · p. 6
  • Neither workflow nor malleability is a drop-in replacement for applications with fully static resource allocation.Workflows require modular applications and a dedicated workflow language; malleability has a user-friendly interface but requires more complex state management and execution with variable resources.Source: B. Discussion · p. 6

V. CONCLUSION

The paper proposes malleability and workflow-based strategies for hybrid HPC-QC resource allocation and evaluates them with a representative application. Under resource contention, malleability reduces execution time and node-seconds compared with the static baseline, with trade-offs against workflows.

V. CONCLUSION

  • Under resource contention, malleability reduces execution time by approximately ≈44% versus the baseline and ≈8% versus workflow, and node-seconds by ≈54% versus baseline while incurring ≈17.5% overhead versus workflow.These results are reported for the representative hybrid HPC-QC application.Source: §V · p. 6
  • The paper proposes dynamic resource allocation through malleability, alongside a workflow-based approach, for HPC-QC workloads.Malleability had not previously been explored in the HPC-QC context, to the authors’ knowledge.Source: §V · p. 6
  • Future work includes larger-scale execution, more complex resource contention, and possibly an actual quantum machine.Source: §V · p. 6

Related papers

Loading 2508.04217v1…