RASER: Resilient Agent Scheduling and Execution Runtime for HPC Clusters

1 Introduction

• Agentic workflows can span minutes to hours, decompose complex objectives into tasks, use external tools and APIs, and remain computation-intensive.These workloads create demand for high-end computational hardware.Source: §1 · p. 1
• Long-running, fault-prone agents, unpredictable API calls, preemption, heterogeneous task durations, and multi-tenancy make HPC execution difficult.Slurm provides job arrays, requeue, and container integration, but its arrays support static partitioning.Source: §1 · p. 2
• RASER combines Slurm job arrays, shared-filesystem work stealing, application-level checkpointing with Slurm requeue, and Apptainer isolation without image modification.It runs in userspace without kernel privileges or external database infrastructure.Source: §1 · p. 2

2 Related Work

• Robust execution of heterogeneous agent ensembles requires dynamic scheduling, fault tolerance, checkpointing, and secure isolation, yet production HPC environments often preclude software-stack changes and high privileges.The section reviews prior workflow, scheduling, fault-tolerance, and containerization systems as partial solutions.Source: §2 · p. 3
• RADICAL-Pilot, Parsl, Dask, Balsam [14], and TaskVine each lack at least one capability relevant to dynamic, stateful agent workflows.The cited limitations include external database and middleware requirements, static provisioning, or missing Slurm-array integration and application-level checkpointing.Source: §2 · p. 3
• Ray targets distributed AI primitives, while Apptainer, Shifter, and Charliecloud provide HPC containers with differing administrative coupling or feature limits.DMTCP and CRIU offer process-level checkpointing but are not compatible with capturing application-level agent reasoning state; Kubernetes adds overhead and replaces the existing scheduler.Source: §2 · p. 3
• RASER combines Slurm job arrays, requeue, and container support to provide agent-ensemble execution without administrative coordination, persistent services, databases, or kernel privileges.The section states that none of the existing approaches provides the required capabilities out of the box.Source: §2 · p. 3

3 RASER’s Architecture and Implementation

• Plugins connect RASER’s generic orchestrator to agent frameworks by handling setup, injecting variables, updating heartbeats, and adding state-saving hooks when needed.Users can supply workload-specific implementations while keeping the main codebase generic.Source: §3 · pp. 4–5
• Each array task runs a persistent Bash wrapper that launches the Python orchestrator in a container and relaunches it after a process crash.The wrapper passes the workspace, queue path, and API keys through bind mounts and environment variables.Source: §3 · p. 5
• RASER uses unmodified user containers with a runtime bind-mounted at execution and gives each agent a private workspace to prevent filesystem cross-contamination.Framework settings and API keys are injected at runtime.Source: §3 · p. 5
• On preemption or wall-time expiration, agents serialize and archive their workspace and state, exit with code 99, and resume from the checkpoint after Slurm requeues the job.The shared filesystem also stores checkpoints, logs, task queues, and results.Source: §3 · p. 6
• Agents dynamically claim or reclaim queued tasks using shared-filesystem polling and atomic file locking, decoupling task assignment from Slurm array indices.Peers may reclaim tasks whose timestamps have gone stale for the configured interval, three minutes by default.Source: §3 · p. 6
• Resource-aware tiers route tasks with elevated failure counts toward higher-memory agents, while lower-tier agents prioritize fresh tasks.This policy is intended to keep lightweight tasks from monopolizing high-memory agents.Source: §3 · p. 6
• Task failures can trigger retries, re-queuing, or reassignment, while job-level failures may use Slurm requeue; when checkpointing fails, lost computation must be reexecuted.The implementation assumes Apptainer 1.1+ with user namespaces, a POSIX filesystem supporting fcntl, Slurm 20.02+ with requeue and signal support, and Python 3.8+.Source: §3 · pp. 7–8

• RASER combines non-coordinated multi-tier load balancing with a user-space, installation-free execution environment that requires no privileged access.Source: §4 · p. 8
• The comparison used 60 SWE-bench-Lite tasks with mini-SWE-agent; direct comparisons against other frameworks were not possible.Source: §4 · p. 8
• Nearly 39% lower makespan than static partitioning on heterogeneous workloads, with near-full CPU utilization.Source: Figure 4 · p. 9, §4 · p. 9
• Under random SIGTERM fault injection, agent states were restored and agents completed all tasks, though lost-computation intervals were not reported.Source: §4 · p. 9
• Checkpoint/restore overhead was under 30 seconds on NVMe-backed filesystems, but larger-scale lock-contention testing remains future work.Source: §4 · pp. 9–10

• Evaluations show reduced makespan compared with static partitioning while achieving near-full CPU utilization.Source: §5 · p. 10
• RASER combines Slurm job arrays and shared-filesystem work stealing with application-level checkpointing and container-based isolation.Source: §5 · p. 10
• Its installation-free, user-space design requires no kernel privileges or external database infrastructure.Source: §5 · p. 10
• Future work includes model switching, distinguishing reasoning from hung agents, and GPU provisioning; current evaluations use CPU-only allocations making external API calls.Source: §5 · p. 10