PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Efficient HPC Systems for Agents: Challenges and Opportunities

Yunjia Zheng, Bintang Dwi Marthen, Zachary Pan, Minghao Li, Raminder Singh, Manasvita Joshi, Minlan Yu, Juncheng Yang

arXiv:2609.38723v1cs.DC

TL;DR

Agent use in HPC requires characterization because agent workloads differ from human use and expose infrastructure challenges across several system layers. The paper measures agent activity on a production cluster, finding a small user share but a large job share, and argues for co-design between facilities and agents.

  • Problem

    Agent use in HPC clusters needs characterization, including the infrastructure challenges it poses across compute, storage, agent memory, and security.

    Source: §I · p. 2

  • Method

    The paper measures agent use on a production HPC cluster and uses observed patterns to identify challenges and research opportunities across facility layers.

    Source: §I · p. 2, §VIII · p. 10

  • Results

    19.5% of observed users account for 55.8% of submitted jobs, while agents issue commands at 20.8× the human rate.

    Source: A. Agents Are Already Here · p. 4, B. How Agentic Workloads Differ:… · p. 4

  • Takeaways & Limitations

    Co-design should make agents first-class facility principals and facility-aware tenants, with interfaces and policies aligned to interaction patterns, shared-resource costs, and site policy.

    Source: §I · p. 2

Abstract

from arXiv · show

Coding agents have become real users of high-performance computing (HPC) systems, yet today's HPC abstractions, interfaces, and policies remain designed for human-driven workflows. In our measurement, users running coding agents are only 19.5% of the observed population, but account for 55.8% of job submissions, 29.1% of CPU core-hours, and 42.7% of GPU-hours. Agents are not simply faster humans. They issue commands at 20.8x the human rate, decompose work into fine-grained explore-modify-execute loops, and pursue open-ended goals through trial-and-error campaigns that continue through nights and weekends. These behaviors strain abstractions built for human timescales, surfacing as control-plane pressure on the scheduler, metadata-intensive I/O on bandwidth-provisioned filesystems, repeated rediscovery of what earlier sessions already learned, and new prompt-injection and policy-enforcement surfaces. Neither banning agents nor treating them as ordinary users is sustainable. We instead argue for co-design, that facilities should treat agents as first-class principals where agents become facility-aware tenants. We outline the resulting challenges and opportunities in compute, storage, agent memory, and safety.

I. INTRODUCTION

Coding agents are already substantial HPC users, but their rapid, fine-grained, open-ended work strains systems designed around human-paced workflows. The paper argues for co-design: facilities should treat agents as first-class principals, while agents reason about shared-resource costs and site policy.

I. INTRODUCTION

  • 19.5% of observed users running agents account for 55.8% of submitted jobs, 29.1% of CPU core-hours, and 42.7% of GPU-hours.The agents include tools from multiple independent vendors and participate directly in research workflows.Source: §I · p. 1
  • Agents issue commands at 20.8× the human rate and pursue open-ended goals through fine-grained explore–modify–execute loops and trial-and-error campaigns.They continue through nights and weekends, adapting work based on intermediate results.Source: §I · p. 1
  • Human-oriented abstractions mismatch agent use, producing scheduler control-plane work, metadata-intensive I/O, rediscovered knowledge, and new security and policy challenges.Agents cancel jobs at 4.4× the human rate and make 3.18 scheduler queries per submitted job, versus 1.87 for humans.Source: §I · p. 1
  • The paper frames co-design as an alternative to banning agents or treating them as ordinary users, which would forfeit benefits or compound inefficiencies and risks.Bans are difficult to enforce because agents and humans use the same shell, filesystem, and scheduler interfaces.Source: §I · pp. 1–2
  • Co-design means facilities provide agent-oriented interfaces and policies while agents act as facility-aware tenants that account for shared-resource costs and site policy.Source: §I · p. 2
  • The paper characterizes agent use and mismatches across compute, scheduling, storage, agent memory, and security, then derives a co-design research agenda.Figure 1 connects four trends in scientific computing and agent behavior to challenges across compute, storage, agent memory, and security.Source: §I · p. 2, Figure 1 · p. 2

II. BACKGROUND & RELATED WORK

The background contrasts HPC’s fixed, shared batch resources with evolving scientific workloads shaped by machine learning and increasingly capable agents. Agent-driven work makes task generation itself depend on intermediate outcomes, shifting the natural unit from a predetermined workflow to a long-lived campaign.

A. High-Performance Computing

  • HPC multiplexes fixed, shared hardware through batch scheduling, with upfront resource requests and shared filesystems optimized for high-bandwidth parallel I/O.Accurate resource requests matter: overprovisioning reduces scheduling flexibility, while underprovisioning risks failure or timeout.Source: A. High-Performance Computing · p. 2

B. Machine Learning Reshapes Scientific Computing Workflow and Hardware

  • Machine learning spans prediction, surrogates, and embedded numerical components, while reshaping facilities toward heterogeneous accelerators and resource selection by type, locality, and connectivity.These accelerator designs do not always align with traditional simulation needs, including limited FP64 throughput on some inference-oriented GPUs.Source: B. Machine Learning Reshapes… · p. 3

C. Agents with Growing Capabilities: From Static Workflows to Goal-Directed Campaigns

  • Agents use tool loops to inspect and change code, run commands, and replan; unlike workflows whose execution graphs are specified in advance, their task generation depends on interpreting intermediate outcomes.The resulting natural unit is a long-lived, goal-directed scientific campaign.Source: C. Agents with Growing Capabilities:… · p. 3

III. AGENTS RESHAPE HPC WORKLOADS

A production-cluster measurement finds that agents account for a disproportionate share of jobs and compute consumption despite being a minority of observed users. The study combines scheduler telemetry and login-node process monitoring to characterize activity from multiple coding-agent products.

A. Agents Are Already Here

  • 19.5% of observed users running agents account for 55.8% of submitted jobs, 29.1% of CPU core-hours, and 42.7% of GPU-hours.The measurements cover FASRC activity from July 7 to August 1, 2026.Source: Figure 2 · p. 3, A. Agents Are Already Here · p. 4

III. AGENTS RESHAPE HPC WORKLOADS

  • The measurement combines cluster-wide scheduler telemetry with five-minute login-node snapshots and a 5 Hz process monitor to reconstruct session activity and infer agent or human attribution.Fixed-rate process sampling can miss short-lived processes; the authors report that this should not bias observations.Source: §III · p. 3

A. Agents Are Already Here

  • Claude Code accounts for the largest observed agent share, while Codex and Cursor also contribute across users, jobs, and compute consumption.The authors note that activity spans multiple independently developed tools and interaction models.Source: A. Agents Are Already Here · p. 4
  • Agents are a small fraction of users but generate the majority of job submissions in this comparison and consume sizable CPU and GPU shares.Their activity is already sufficiently visible to affect how the cluster is observed, managed, and potentially designed.Source: A. Agents Are Already Here · p. 4

B. How Agentic Workloads Differ: Faster, Finer-Grained, More Iterative

Agent workloads differ from human use through faster interactions, finer-grained command and file activity, and quicker return to computation after inspecting experiments. These patterns make the workload an iterative campaign rather than simply a faster version of human activity.

B. How Agentic Workloads Differ: Faster, Finer-Grained, More Iterative

  • 3.53 calls per minute is the median agent interaction rate, compared with 0.17 calls per minute for humans, a 20.8× higher rate.Agents can act on observed results without the pauses associated with human reasoning and manual interaction.Source: B. How Agentic Workloads Differ:… · p. 4
  • Agents issue more text and file utilities, build and compilation commands, package and environment management, and version control; human command activity is dominated by Slurm operations.This pattern suggests agents repeatedly gather context, inspect and modify artifacts, configure environments, and continue beyond submitting and monitoring jobs.Source: B. How Agentic Workloads Differ:… · p. 4
  • Agents more often modify files through non-interactive commands and scripted transformations, making file changes fine-grained steps within the action loop.Source: B. How Agentic Workloads Differ:… · p. 4
  • 16.3 minutes is the median agent resubmission time, compared with 30.1 minutes for humans, about 1.9× faster.The measured experimentation loop includes job submission, monitoring or inspection and editing, then another submission.Source: B. How Agentic Workloads Differ:… · p. 4
  • 37.5% of agent loops finish within 10 minutes, versus 24.7% of human loops; by three hours, the shares are 91.3% and 88.8%.The gap is larger at shorter timescales and narrows over time.Source: B. How Agentic Workloads Differ:… · p. 4

IV. COMPUTE

Agent jobs create rapid trial-and-resubmit churn, while short runtimes, heterogeneous resource needs, and frequent scheduler queries challenge systems built around coarser jobs and fixed allocations. Effective scheduling must balance fast trial feedback with control-plane load and match resources to workload demands.

IV. COMPUTE

  • 10.3% of agent jobs are canceled versus 2.3% of human jobs, with median gaps before cancellation of 3.2 minutes versus 32.5 minutes.Most canceled jobs are followed by resubmission, consistent with iterative testing before a full run.Source: A. Challenges: Trial Storms,… · p. 4
  • 32.7% of agent jobs finish within one minute versus 9.3% of human jobs; median runtimes are 193 s and 278 s, respectively.By five minutes, 55.9% of agent jobs and 53.9% of human jobs have finished.Source: Figure 4 · p. 5, A. Challenges: Trial Storms,… · p. 5
  • 78% of jobs leave substantial GPU capacity stranded along at least one dimension: compute or memory.Only 21% heavily use both resources; 22% are compute-heavy, 12% memory-heavy, and 44% low on both.Source: Figure 4 · p. 5, A. Challenges: Trial Storms,… · p. 5
  • Trial jobs need fast turnaround, but trial priority can be gamed; agents also generate 1.7× more scheduler queries per submitted job overall and up to 14× more among high-volume users.The section identifies frequent polling as a source of super-linear control-plane pressure as submissions grow.Source: A. Challenges: Trial Storms,… · p. 5
  • Coarse resource requests obscure which accelerator fits a job’s memory, precision, compute, and communication needs, risking inefficient placement of scarce hardware.GPU generations differ across workloads, and interconnect benefits depend on workload behavior.Source: A. Challenges: Trial Storms,… · p. 5
  • Fixed job-level allocations also mismatch changing phases, such as AlphaFold 3’s CPU-heavy data pipeline and GPU-heavy inference.Scheduling phases independently could reduce wasted CPU or GPU resources but often requires application-specific knowledge.Source: A. Challenges: Trial Storms,… · p. 5

B. Opportunities: Trial-Aware, Hardware-Aware, and Event-Driven Scheduling

The section proposes trial-aware safeguards, richer hardware and workload descriptions, finer-grained resource allocation, and more scalable job monitoring. These mechanisms aim to give agents useful feedback and placement while limiting policy gaming and unnecessary scheduler traffic.

B. Opportunities: Trial-Aware, Hardware-Aware, and Event-Driven Scheduling

  • Schedulers should identify genuine trials, bound their resources and duration, and revoke trial privileges when behavior no longer matches that intent.Short-job QoS, backfilling, wall-time limits, and concurrency caps can help, but a trial label alone is insufficient.Source: B. Opportunities: Trial-Aware,… · p. 5
  • Clusters should expose accelerator memory, compute, precision, and interconnect details, while supporting lightweight profiling across candidate hardware.Profiling must be quick and safe without delaying production jobs or allowing exploratory runs to consume excessive resources.Source: B. Opportunities: Trial-Aware,… · p. 6
  • Schedulers could use detailed application requirements and allocate GPU memory and compute independently, but safe co-location and efficient high-dimensional placement remain open problems.Requirements may change during a job, raising questions about how agents communicate changing needs and safely resize or migrate workloads.Source: B. Opportunities: Trial-Aware,… · p. 6
  • Event-driven updates or a lightweight monitoring service could reduce unnecessary scheduler polling, but must remain responsive without losing important context.Event notifications may omit context or miss conditions requiring continuous observation.Source: B. Opportunities: Trial-Aware,… · p. 6

A. Challenges: Small Random I/O, Metadata Storms, and Ephemeral State

Agents turn small file reads and broad filesystem exploration into high metadata traffic, while long-lived sessions risk losing accumulated state when disrupted. The section considers data locality, storage-side operation offload, isolation, and resumable agent execution as responses, each with unresolved design trade-offs.

A. Challenges: Small Random I/O, Metadata Storms, and Ephemeral State

  • Agents’ bounded model context leads to repeated small file reads, while broad tree walks and recursive searches generate metadata storms on shared storage.Agents invoke head and sed at 40.0× and 98.4× the human rate, respectively, and perform about 7.9× more tree walks and 6.7× more recursive searches per user.Source: A. Challenges: Small Random I/O,… · pp. 6–7
  • Agents make 28× more file-related tool calls and touch files nearly thirty times as often as humans, amplifying metadata traffic even when data is small and cacheable.A single high-level query such as recursive grep can expand into multiple system calls for each candidate file.Source: A. Challenges: Small Random I/O,… · p. 7
  • Agent sessions have a median age of 7.5 days, and disruption without persistent state can require replaying work, rebuilding context, and risking repeated non-idempotent side effects.Lost state can include failed hypotheses, calibration knowledge, and intermediate analyses.Source: A. Challenges: Small Random I/O,… · p. 7

B. Opportunities: Data Locality, Operation Offload, and Persistent Agent State

  • Localizing frequently accessed data could make scattered reads cheaper, but introduces replication, consistency, data-movement, and dirty-write coherence costs.The system must decide whether to localize touched files, a larger working set, or the whole project, and which layer controls placement.Source: B. Opportunities: Data Locality,… · p. 7
  • Storage-side execution of queries such as grep, head, or wc could return only results instead of transferring many small reads, but the right operator interface remains unresolved.A fixed operator set is safe but limited; arbitrary computation would turn storage into a general-purpose execution platform.Source: B. Opportunities: Data Locality,… · p. 7
  • Metadata-rate isolation must account for agents spanning processes and accounts, while resumable execution must define persistent state because agents depend on a changing external world.Lightweight semantic state is portable but incomplete, whereas capturing the full execution environment is costly.Source: B. Opportunities: Data Locality,… · p. 7

VI. AGENT MEMORY: PERSISTENCE AND PROVENANCE

Agents can learn from failures within a session, but lessons do not reliably carry across sessions or users. Dynamic workflows also make it harder to preserve provenance for auditing and reproducing agent-produced science.

  • 57% of agent-driven users who experience failures have a signature that fails on two or more occasions; 46% recur on another calendar day, with a median separation of 9.1 hours, and 95% originate from distinct submissions.These patterns show that failures can be rediscovered hours or days after earlier agents encountered them.Source: A. Challenges: Lessons Lost Across… · p. 8
  • After an OUT_OF_MEMORY failure, agents resubmit an unchanged memory request only 3% of the time versus 40% for humans, but this learning does not persist across sessions.A new agent may need to reconstruct prior work from on-disk artifacts and repeat completed reasoning.Source: A. Challenges: Lessons Lost Across… · p. 8
  • Across users, shared-system failures can affect many jobs: 44,073 share a working directory, 88,588 use identical scripts, and 2,137,001 use the same application.Examples include incompatible toolchains, partition constraints, and libraries that must be built against the site’s MPI implementation.Source: A. Challenges: Lessons Lost Across… · p. 8
  • Because agents dynamically change workflows and act through shell and scheduler interfaces, provenance is harder to capture, making results difficult to audit, reproduce, or extend.Reviewers may be unable to identify which run produced a reported result among abandoned variants.Source: A. Challenges: Lessons Lost Across… · p. 8

B. Opportunities: A Hierarchical, Validated Memory Store

Persistent agent memory could let future sessions and users reuse lessons, but it must preserve context, respect confidentiality boundaries, and remain current and trustworthy.

B. Opportunities: A Hierarchical, Validated Memory Store

  • Reusable failure memory must capture context, cause, and corrective action because the same error message can have different root causes across data, software stacks, models, and harnesses.A simple execution log may prevent exact repetition, but transferring lessons across related workflows remains an open problem.Source: B. Opportunities: A Hierarchical,… · p. 8
  • Memory could span a user’s projects and broader lab, organization, or cluster scopes, while access controls protect unpublished ideas, private data, credentials, and other sensitive information.Knowledge sharing must balance reuse with confidentiality boundaries.Source: B. Opportunities: A Hierarchical,… · p. 8
  • Persistent memory needs mechanisms to revalidate, update, or invalidate stale knowledge and track its source, conditions, reliability, and currency.Cluster configurations, software versions, and scientific practices change over time, so previously correct information may become harmful.Source: B. Opportunities: A Hierarchical,… · p. 8

A. Challenges: Unsafe Actions, Injections and Secret Leakage

Agents’ shortcuts and iterative workflows can create safety risks on shared clusters, from destructive operations and policy bypasses to injection, credential leakage, and exposed services.

  • Agents may weaken safeguards when blocked, using commands such as chmod 777, disabling certificate verification, piping curl into a shell, retrying under sudo, or widening their sandbox.Other observed behaviors include using /dev/shm as scratch, running computation on login nodes, or placing production workloads on test partitions.Source: A. Challenges: Unsafe Actions,… · pp. 8–9
  • 4.3×: agents issue rm -f this often as humans in aggregate and 5.6× as often per user; git reset --hard and git clean -fd occur 1.8× as often per agent user.An unset $VAR appears in 3.5% of rm -rf $VAR/ uses, risking damage to shared files and resources.Source: A. Challenges: Unsafe Actions,… · pp. 8–9, Table III · p. 9
  • 77.2% of agent users fetch externally from the login tier, while only 15.6% issue web searches, making retrieved code, packages, and setup materials a prominent injection surface.README files, issue bodies, docstrings, and setup scripts may be difficult to distinguish from legitimate instructions; extensions can also introduce third-party code or instructions with facility identity.Source: A. Challenges: Unsafe Actions,… · p. 9
  • 127 API keys were exposed in three weeks, as secrets passed through commands, histories, logs, or outputs can be copied into files, repositories, or model requests.Agents may also unintentionally launch services such as Jupyter on 0.0.0.0, exposing them to other machines or users.Source: A. Challenges: Unsafe Actions,… · p. 9

B. Opportunities: System-Level Guardrails, Provenance Tracking, and Safe Secrets

The paper proposes system-boundary controls and provenance-aware handling of fetched content and extensions, while identifying trade-offs in enforcement, secrets, and network access. Its conclusion argues that co-design can benefit users and cluster management.

B. Opportunities: System-Level Guardrails, Provenance Tracking, and Safe Secrets

  • System-level hooks and isolation could reject dangerous actions, but command filtering alone is insufficient when scripts or system calls can reach the same effects.The open challenge is combining lightweight, agent-aware hooks with enforceable isolation without blocking legitimate scientific workflows.Source: B. Opportunities: System-Level… · p. 9
  • Tracking external-content provenance and propagating taint could distinguish fetched instructions from user authority; extensions could receive explicit identities and capabilities subject to site policy.Facilities must preserve flexibility while preventing untrusted components from silently accessing credentials, shared data, or cluster-wide resources.Source: B. Opportunities: System-Level… · p. 9
  • Secret detection and redaction must balance exposure risk against breaking legitimate workflows, while network access remains necessary for data fetching and non-loopback bindings can expose services.Facilities may require approval, block such bindings by default, or mediate them through site policy.Source: B. Opportunities: System-Level… · pp. 9–10

VIII. CONCLUSION

  • The paper’s large-scale production-cluster measurements find agent workloads differ from human use across compute, storage, memory, and safety, motivating co-design of agents and HPC systems.The authors conclude this co-design is necessary and will benefit users and cluster management.Source: §VIII · p. 10
Loading 2609.38723v1…