PaperlayerView exact revision on arXiv

Source-linked AI summary

Diamond Agent: Agentic Control of Federated HPC Resources as a Service

Haotian Xie, Junlin Chen, Mingkai Zheng, Yifan Zhu, Minu Mathew, Max Burnette, Yadu Babuji, Volodymyr Kindratenko, Shivaram Venkataraman, Kyle Chard, Ian Foster, Zhao Zhang

arXiv:2609.06181v1cs.DCcs.AI

TL;DR

Diamond Agent addresses fragmented workflow context and execution across independently administered, heterogeneous supercomputers. It uses a unified agent-facing interface with typed skills, live-state placement, and event-driven continuation; in matched multi-site experiments, live-state policies reduced typical additional completion time to four seconds.

  • Problem

    Managing jobs and data across heterogeneous, independently administered supercomputers is difficult because workflow context and execution state are fragmented.

  • Method

    Diamond Agent uses an agent-facing HPC execution contract with typed skills, stable task handles, unified results, shared context, resource envelopes, live federation state, and event-driven continuation.

  • Results

    Four seconds is the typical additional completion time achieved by both live-state policies, compared with 1,141 seconds for the fixed-system mean additional completion time.

  • Takeaways & Limitations

    Live federation state can avoid the large penalty of always using one system, while the current estimator often identifies near-best placements without guaranteeing minimum actual completion time.

  • Takeaways & Limitations

    The current placement model uses hand-designed estimators rather than continuously learning from previous executions, and resource sizing is separated from placement.

Abstract

from arXiv · show

Efficiently aggregating and orchestrating computing power across heterogeneous clusters for HPC workflows faces four practical challenges: preserving workflow context across independently administered clusters, moving large datasets between sites, reasoning about site-specific environments and scheduler policies, and exploiting live queue and resource states for efficient task scheduling. To this end, we design Diamond Agent, an agentic system that enables intelligent execution of HPC workflows across heterogeneous clusters with typed skills as the interface. Diamond Agent provides an agent-facing workspace and skills that unify cross-site resource discovery, resource specification, data movement, task execution, and result retrieval. A centralized Diamond Agent instance can operate multiple supercomputers without being deployed separately on each login node. Diamond Agent translates high-level agent actions into valid site-specific executions, moves data through Globus Transfer, and uses live system capability and queue information to select feasible placements. Its event-driven continuation mechanism decouples agent actions from long-running batch jobs: persistent services monitor remote execution and resume the agent only when a result or decision-relevant event is available. We experiment with 27 hours of telemetry and 19 matched multi-site submission rounds comprising 83 jobs across four production supercomputers. Compared with a fixed-site baseline, Diamond Agent reduces the median additional completion time relative to the fastest observed placement from 42 seconds to 4 seconds, a 10.5x reduction.

I. INTRODUCTION

Diamond Agent presents a unified, agent-facing control plane for operating workflows across heterogeneous supercomputers. Typed skills, persistent cross-site context, event-driven continuation, and deterministic placement address fragmented execution and long-running remote jobs.

  • The system unifies runtime configuration, job submission and monitoring, result management, and large-file synchronization through authorized Globus Compute endpoints and Globus Transfer.
  • ResourceEnvelope specifies minimum and maximum useful resources plus compatibility constraints, while deterministic rules and best-feasible selection produce a valid site and queue.
  • Diamond Agent lets one agent operate multiple supercomputers through typed skills, stable task handles, unified result manifests, and shared cross-site context.
  • Event-driven continuation lets the agent become inactive during remote execution and resume from durable results, milestones, or unresolved exceptions.
  • The implementation provides persistent, restart-safe task and group orchestration with site-specific execution, bounded adaptation, and agent continuation across four production supercomputers.

II. BACKGROUND AND RELATED WORK

The paper positions Diamond Agent as a cross-site execution layer for workflows spanning independently administered systems. It combines durable federation with live resource awareness because queue length alone does not capture allocatable capacity or placement feasibility.

  • Diamond Agent builds a common control plane above user-authorized Globus Compute endpoints while leaving underlying schedulers independently administered.
  • Unlike Kubernetes or cloud brokerage, its ResourceEnvelope maps to one fixed HPC allocation and candidates are constrained by site policy, queue state, node shapes, and per-site credits.
  • A single workflow may require CPU preprocessing, GPU training, and evaluation across systems with different accounts, partitions, accelerators, and storage paths.
  • Queue length alone is insufficient: DeltaAI had a median 823 pending jobs and 35 free devices, while Anvil had fewer pending jobs but one free device.
  • Persistent control is required because batch jobs outlive active agents and queued work may become better placed on another authorized system.

IV. AGENTIC INTERACTION MODEL

Diamond Agent models workflows as typed actions separated by asynchronous remote execution. A persistent control plane stores context, translates site-specific details, and resumes the agent only when new events require reasoning.

  • The agent follows an observe–plan–act–yield–resume cycle, invoking typed skills before yielding while the control plane executes remote actions.
  • The agent uses site-independent identifiers, while Diamond translates accounts, partitions, QoS fields, scheduler commands, endpoints, and paths into site-specific executions.
  • Submission returns immediately with a durable handle and placement trace, while result access returns a compact provenance manifest before large artifacts are fetched.
  • After a registered condition, the Task Orchestrator appends a stable event and reactivates the agent with a compact task or group payload.
  • Persistent storage preserves objectives, resource envelopes, placement decisions, provenance, sibling states, attempts, and result locations outside the model context window.
  • Deterministic rules enforce authorization, policy, credits, compatibility, retry budgets, and blocked operations, allowing routine recovery without model invocation.

V. SCHEDULING FORMULATION

Diamond Agent formulates scheduling around a typed ResourceEnvelope, deterministic feasibility rules, and best-feasible placement over current resource snapshots. The policy selects a concrete fixed batch allocation while preserving authorization, compatibility, capacity, credit, and queue constraints.

  • A. ResourceEnvelope and Candidate Set: A ResourceEnvelope specifies minimum and maximum GPU, CPU, memory, node, and wall-time resources plus architecture, accelerator, container, data, and policy constraints.It may be user-authored or AI-drafted, but drafts remain subject to schema validation and deterministic scheduling rules.
  • A. ResourceEnvelope and Candidate Set: Independent resource ranges are converted into one fixed batch allocation before submission.The implementation chooses a single concrete request rather than dynamically varying resources during execution.
  • A. ResourceEnvelope and Candidate Set: The scheduler represents each candidate as a system, partition, account, and concrete job size selected from the envelope.Candidate feasibility depends on the current ResourceSnapshot and deterministic rule checks.
  • A. ResourceEnvelope and Candidate Set: Feasible candidates must pass endpoint, authorization, partition, architecture, accelerator, capacity, wall-time, container, and user-limit checks.The predicted charge must also fit the user’s site-specific budget, which is not treated as globally fungible.
  • B. Best-Feasible Placement: The selected candidate is determined after feasibility filtering and objective evaluation, with deterministic tie-breaking when estimates are equal.The supplied passages define the selection policy but do not provide the completed candidate expression.
  • B. Best-Feasible Placement: The default earliest-completion objective estimates completion as queue delay plus runtime plus environment-preparation cost.The queue-delay estimate uses request-sized free GRES or CPU capacity and then queue depth; ties consider credit, specialized-resource waste, and user preference.
  • B. Best-Feasible Placement: For a fixed snapshot and feasible candidate set, the policy enumerates candidates and chooses the one with minimum estimated completion time.This is a best-feasible policy rather than a guarantee of minimum actual completion time or global optimality across interacting or future tasks.

VI. SYSTEM DESIGN

The system design separates agent-facing operations from federated infrastructure services that maintain resource state, enforce placement decisions, and manage asynchronous execution. It also integrates data movement through durable references and transfer handles rather than placing large artifacts in the agent context.

  • Architecture: The architecture separates an agent-facing control loop from an infrastructure control loop for typed skills, resource placement, asynchronous execution, and site-specific actuation.This lets the agent reason over one logical workspace while deterministic services handle long-lived infrastructure state.
  • Agent Gateway: The Agent Gateway maps actions into typed operations and stores objectives, envelopes, traces, attempts, event cursors, and result references outside model context.Submission returns control metadata while logs and large artifacts remain referencable rather than being injected automatically.
  • Global Resource View: The Global Resource View maintains static system and policy metadata alongside dynamic endpoint, node, accelerator, CPU, queue, and timestamp fields.Degraded endpoints remain visible, but stale or offline records cannot pass placement checks.
  • ResourceEnvelope: ResourceEnvelope drafting can use task text and code context, but the scheduler remains model-free and deterministically records sizing, rejection, estimates, credits, and tie-breaks.The draft is a typed proposal rather than a placement decision.
  • Data Movement: Diamond Agent inherits Globus Transfer for large-file movement using logical data references that resolve to authorized collections and paths.Inputs can be staged before dispatch, and outputs can be returned or replicated for later tasks.
  • Data Movement: Transfers return durable handles, are monitored outside the model loop, and expose compact status or artifact references with provenance.Result manifests provide locations and metadata instead of inserting large datasets, checkpoints, or images into the model context.

D. Event-Driven Orchestration and Task Groups

Persistent orchestration converts placements into durable task and group histories while allowing the agent to yield during remote execution. Registered results, milestones, and exceptions reactivate the agent with compact, deduplicated continuation state.

  • Task Orchestration: The Task Orchestrator records scheduler identifiers, endpoint bindings, state transitions, retry budgets, continuation registrations, and events for each durable task.Failed or pending work may be retried or re-placed under explicit policies, while attempts remain separately tracked.
  • Task Groups: Task groups coordinate independently placed children that share objectives, envelopes, rounds, and policies.The orchestrator aggregates child states, can relax harmful soft constraints, and advances later rounds after required children become terminal.
  • Continuation: The service exposes task, group, result, replan, cancel, and event interfaces, returning a handle once the scheduler accepts remote dispatch.An event bridge deduplicates durable events and supplies compact continuation payloads instead of requiring continuous agent activity.
  • Continuation: Continuation payloads contain identifiers, new state, placement and attempt provenance, result manifests, and bounded next actions.Large outputs remain outside the prompt until the agent requests the records needed for interpretation.
  • Control Service: Diamond Agent runs as one control service that refreshes federation state and advances tasks and groups against each resulting snapshot.Globus Compute delegates remote actions while each center retains its scheduler and access boundary.
  • Persistence: Persistent records allow startup recovery and independent lifetimes for the agent runtime, control service, and remote batch jobs.Pending and running jobs are rebound to original endpoints, while scheduler accounting remains authoritative for final state and timing.

C. Globus Transfer Integration

The design integrates persistent Globus Transfer with live, constraint-aware placement and site-specific submission. Resource profiles and probes inform deterministic validation, sizing, queue estimation, retries, and re-placement across heterogeneous systems.

  • C. Globus Transfer Integration: Diamond’s data manager submits stage-in, stage-out, and site-to-site synchronization through Globus Transfer while recording storage paths, transfer identifiers, and compute-task relationships.Transfer progress becomes part of the persistent task timeline and can survive an agent or service restart.
  • C. Globus Transfer Integration: The agent issues a typed transfer intent and later receives compact status or artifact metadata while Globus-managed endpoints move large files.The current placement objective does not jointly optimize transfer duration with queue wait and runtime.
  • Resource Discovery: Each site combines a static profile of hardware and executable policy with live probes of node states, free CPUs, free accelerators, queues, accounts, architecture, and endpoint health.The profile includes partitions, QoS requirements, node shapes, wall-time caps, container support, and scheduler-dialect switches.
  • Resource Discovery: Free GPU capacity is computed from configured and used GRES on usable nodes and merged with profiles into timestamped ResourceSnapshots.This avoids relying on node state alone, which can misrepresent allocatable devices on shared GPU nodes.
  • Resource Discovery: The current site-profile interface is configured beyond the four evaluation systems for NERSC Perlmutter and ALCF Sophia and Polaris.
  • Constraint-Aware Placement: The optional envelope generator produces structured bounds, constraints, objectives, rationale, and confidence, while preserving both the original draft and normalized envelope.Schema validation checks ranges, and the scheduler can operate entirely from a user-authored envelope.
  • Constraint-Aware Placement: Submission removes candidates failing nine deterministic rule categories before sizing or objective evaluation.Rules enforce authorization, partitions, architecture, accelerator and memory requirements, capacity, wall time, containers, user limits, and site-specific credit.
  • Constraint-Aware Placement: For feasible records, sizing starts from maximum useful resources, respects observed and site-allowed capacity, and never falls below minimum viable resources.Wait estimation uses request-sized free GRES or CPU capacity before node-idle state, combines queue pressure, and includes explicit environment-preparation cost when available.

G. Remote Execution, Persistence, and Safety

Diamond Agent evaluates a federated execution substrate that translates resource intent into authorized site-specific jobs while preserving operational state and experiment traceability. The evaluation is organized around resource views, envelope drafting, placement, multi-system execution, and asynchronous continuation.

  • Final wait and runtime are audited from scheduler accounting rather than inferred from polling intervals.
  • The evaluation asks whether Diamond Agent maintains a useful resource view, drafts executable ResourceEnvelopes, selects effective live placements, adapts multi-system execution, and resumes workflows asynchronously.
  • The service evaluates resource specification, live placement, adaptive multi-system execution, and asynchronous continuation across federated HPC systems.

A. Testbed and Method

The evaluation spans four production systems and combines telemetry, envelope-sizing experiments, and matched multi-site submissions. Results characterize observation overhead, AI-assisted resource specification, and live placement quality.

  • Testbed: The testbed spans four systems at three centers, nine partitions, x86-64 and Arm architectures, and five accelerator configurations over 27 hours.
  • ResourceEnvelope sizing: The envelope-sizing study generated three AI ResourceEnvelopes without manual edits and ran scaling probes totaling 33 completed Slurm jobs.
  • Matched submissions: The matched-submission campaign comprised 19 completed rounds and 83 submissions across seven CPU, six one-GPU, and six four-GPU rounds.
  • Telemetry: The collector produced 543 ResourceSnapshots and 4,737 partition records, with 65.6 seconds median refresh duration and 183.3 seconds mean data freshness.
  • Telemetry: The resource view maintained minute-scale observations across four independently administered systems with modest overhead, supporting meta-scheduling decisions evolving over minutes.
  • ResourceEnvelope sizing: All three AI drafts separated CPU-only, one-GPU inference, and one-to-four-GPU DDP workloads, and every derived request passed the production rule chain.
  • ResourceEnvelope sizing: AI envelopes remained auditable hypotheses because useful scaling and completion-optimal sizing differed under batch queues.
  • Safety: An intentionally impossible request for at least 200 GB of memory per accelerator was rejected at every site with per-candidate explanations.

D. RQ3: Matched Multi-Site Placement

Matched submissions compare deployable placement policies against the fastest observed candidate using scheduler-accounted outcomes. Best-feasible substantially reduces the fixed-site penalty but retains large tail cases and does not guarantee minimum actual completion time.

  • Policies: Best-feasible evaluates all feasible candidates and minimizes estimated completion time using queue delay, runtime, and environment-preparation estimates.
  • Results: The fixed-site policy incurred 1,141 seconds mean additional completion time and 5,873 seconds at p95.
  • Results: Both live-state policies reduced typical additional completion time to 4 seconds and selected the actual fastest candidate in 47% of rounds.
  • Limitations: Best-feasible retained a 536-second mean additional completion time and 3,000-second p95, with a few four-GPU placements raising its mean above shortest queue.
  • Limitations: The live federation avoided the large penalty of always using one system, but the estimator was not accurate enough for larger resource requests and did not guarantee minimum actual completion time.

F. RQ5: End-to-End Agent Continuation

Diamond Agent demonstrates end-to-end agent continuation across production supercomputers while separating active agent reasoning from persistent remote execution. The design also identifies remaining gaps in placement optimization, online learning, and workflow-level controls.

  • End-to-end continuation: Two coding agents used the common skill interface to deploy and fine-tune workloads, then resumed from completion events and result manifests.The workflows targeted one A40 GPU at NCSA Delta and one GH200 at NCSA DeltaAI.
  • Event-driven continuation: Neither workflow polled Slurm during asynchronous execution; the control service monitored jobs and delivered terminal states and result manifests upon reactivation.A five-minute model-side polling comparison would require 8 and 17 status calls over the measured intervals, while the authors characterize this comparison as analytical.
  • Limitations and future work: The current implementation separates resource sizing from placement and relies on hand-designed estimators rather than continuously learning from previous executions.Resource shape, queue delay, site choice, and data-movement cost are not jointly optimized in the current placement model.
  • Limitations and future work: Future work proposes online-learning-based scheduling, joint optimization of sizing and placement, and workflow-level auditing and risk controls.The proposed controls would account for model tokens, node-hours, accelerator-hours, and cumulative resource consumption.
  • System scope: Diamond Agent unifies cross-site context, data movement, resource observation, placement, persistent execution, and result delivery through one agent-facing interface.Remote compute and data movement continue under persistent system control while the agent remains inactive until results or decision-relevant events require further reasoning.
Loading 2609.06181v1…