PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Loosely-Coupled Programming on Petascale Systems

Ioan Raicu, Zhao Zhang, Mike Wilde, Ian Foster, Pete Beckman, Kamil Iskra, Ben Clifford

arXiv:0808.3540v2cs.DC

TL;DR

The work addresses how to run highly parallel computations of loosely coupled serial jobs efficiently on petascale systems, broadening the applications that can use them. The approach combines multi-level scheduling, efficient task dispatch, and caching to reduce demands on shared infrastructure. Throughput reached 3,071 tasks/sec on Blue Gene/P with 640 dispatchers and 163840 CPUs; sustained utilization was 95% for DOCK6 on 128K cores and 99.6% for DOCK5 on 116K CPU cores.

  • Problem

    The work addresses how to run highly parallel computations of loosely coupled serial jobs efficiently on petascale systems, broadening the applications that can use them.

    Source: §I · p. 1, B. Hypothesis · p. 2

  • Method

    The approach combines multi-level scheduling, efficient task dispatch, and caching to reduce demands on shared infrastructure.

    Source: §III · p. 3

  • Results

    Throughput reached 3,071 tasks/sec on Blue Gene/P with 640 dispatchers and 163840 CPUs; sustained utilization was 95% for DOCK6 on 128K cores and 99.6% for DOCK5 on 116K CPU cores.

    Source: C. Falkon Task Dispatch Performance · p. 6, A. Molecular Dynamics: DOCK · p. 9

  • Takeaways & Limitations

    Falkon enabled loosely coupled applications to run on Blue Gene/P without application modifications except recompilation.

    Source: B. Hypothesis · p. 2

    Shared-file-system performance remains a bottleneck; future work would transfer files directly between compute nodes over specialized networks.

    Source: D. Shared File System Performance · p. 7

Abstract

from arXiv · show

We have extended the Falkon lightweight task execution framework to make loosely coupled programming on petascale systems a practical and useful programming model. This work studies and measures the performance factors involved in applying this approach to enable the use of petascale systems by a broader user community, and with greater ease. Our work enables the execution of highly parallel computations composed of loosely coupled serial jobs with no modifications to the respective applications. This approach allows a new-and potentially far larger-class of applications to leverage petascale systems, such as the IBM Blue Gene/P supercomputer. We present the challenges of I/O performance encountered in making this model practical, and show results using both microbenchmarks and real applications from two domains: economic energy modeling and molecular dynamics. Our benchmarks show that we can scale up to 160K processor-cores with high efficiency, and can achieve sustained execution rates of thousands of tasks per second.

I. INTRODUCTION

The introduction frames loosely coupled, task-parallel applications as a way to bring a broader set of workloads to petascale systems, where conventional tightly coupled MPI applications have been the norm. It places the paper’s focus among problem types distinguished by data size and number of tasks.

I. INTRODUCTION

  • Loosely coupled task-parallel workflows exchange data between dependent tasks through files, databases, or other means.The introduction connects their growing use to increasingly complex data analysis and the limits of manual processing and exploration.Source: §I · p. 1
  • The work targets use of petascale systems by applications that can be linked into useful workflows but are not organized as tightly coupled MPI programs.Blue Gene/P systems incorporate high-speed interconnects and features intended to support tightly coupled parallel computations.Source: §I · p. 1
  • Problem types span tightly coupled MPI, analytics, loosely coupled many-task computing, and data-intensive many-task workloads as data size and task count increase.The paper focuses on the many-task category at supercomputer scales, including hundreds of thousands of processors.Source: §I · p. 1

A. Many-Task Computing (MTC)

Many-Task Computing (MTC) describes high-performance computations made up of distinct activities linked through files or messages. The paper distinguishes MTC from related workload labels by its emphasis on large numbers of tasks, heterogeneous work, and applications that need not be embarrassingly parallel.

A. Many-Task Computing (MTC)

  • MTC emphasizes using many computing resources over short periods, whereas HTC typically uses resources for months or years.The respective primary metrics include seconds-scale rates such as FLOPS, tasks/sec, and MB/sec I/O for MTC, versus operations per month for HTC.Source: A. Many-Task Computing (MTC) · p. 2
  • MTC comprises distinct activities coupled through file-system operations or message passing, with tasks varying in size, parallelism, and data or compute demands.Task sets can be static or dynamic, homogeneous or heterogeneous, and loosely or tightly coupled.Source: A. Many-Task Computing (MTC) · p. 2
  • MTC can include complex, communication-intensive applications, unlike the narrower notion of embarrassingly parallel tasks that need little communication.The paper also notes that “workflow” is used broadly and that MTC workloads need not be naturally expressed as MPI programs.Source: A. Many-Task Computing (MTC) · p. 2

B. Hypothesis

The paper hypothesizes that MTC applications can execute efficiently on supercomputers and tests this with Swift and Falkon, microbenchmarks, and real applications. It motivates petascale MTC with I/O capabilities, operating costs, utilization opportunities, and workloads that need petascale computing power.

B. Hypothesis

  • The paper’s hypothesis is that MTC applications can execute efficiently on contemporary supercomputers, and it reports evidence from a 160K-core Blue Gene/P.It states that applications ran without modifications except recompilation.Source: B. Hypothesis · p. 2
  • Microbenchmarks scaled to 160K processor-cores with high efficiency and thousands of tasks per second, while two application domains scaled to 128K cores.The evaluation used Swift and Falkon and included economic energy modeling and molecular dynamics applications.Source: B. Hypothesis · p. 2

C. Why Petascale Systems for MTC Applications?

  • Petascale I/O may suit MTC workloads involving parallel tasks, shared inputs, or substantial communication, although collective I/O was left for future work.The passage contrasts a potential 8GB/s shared-filesystem rate with the more typical 0.1GB/s to 1GB/s range at Grid sites.Source: C. Why Petascale Systems for MTC… · pp. 2–3
  • The paper argues that petascale systems can cost less to manage and run than conventional clusters or Grids, citing Blue Gene/P power efficiency and centralized management.It reports 0.35 GF/watt for a 13.9 TF Blue Gene/P rack, compared with 0.12 GF/watt for average Top500 systems.Source: C. Why Petascale Systems for MTC… · p. 3
  • Petascale MTC can use idle system portions through back-filling, and some applications need its compute power to obtain results within a reasonable timeframe.The paper says this processing can make vast computations possible quickly enough to address scientific challenges or emergencies.Source: C. Why Petascale Systems for MTC… · p. 3

II. RELATED WORK

Prior work explored high-throughput workloads on Blue Gene systems and distributed computing pools, while this paper focuses on efficient MTC execution at petascale. Its implementation combines resource allocation across scheduling levels, high-throughput task dispatch, and caching to reduce shared-infrastructure demands.

II. RELATED WORK

  • Earlier Blue Gene/L HTC approaches reported performance at least an order of magnitude below this paper’s results at small scales, with overheads expected to widen the gap.Those approaches included node rebooting after each task, unlike launching another process.Source: §II · p. 3
  • Distributed HTC platforms target processor pools, while MapReduce is less suited to chaining existing applications and may require custom filtering scripts.The paper contrasts its performance-and-efficiency focus with Condor’s emphasis on robustness and recoverability.Source: §II · p. 3

III. REQUIREMENTS AND IMPLEMENTATION

  • Multi-level scheduling allocates Blue Gene/P resources from Cobalt in 256-core psets, then exposes them at single-core granularity for individual tasks.Without this layer, allocating a full pset to a single-core task could yield utilization of 1/256.Source: §III · p. 3
  • Keeping allocated nodes active amortizes boot costs across many jobs, reducing task launch to loading, running, and returning an exit code.Booting can take up to a thousand seconds when concurrently starting 40K nodes, whereas the lightweight task process can occur in milliseconds.Source: §III · pp. 3–4
  • Falkon streamlines task submission by relying on resource managers for reservations and accounting and client frameworks for recovery and dependencies.The passage reports 3,071 tasks/sec on Blue Gene/P versus 0.5 to 22 jobs per second for traditional local resource managers.Source: §III · p. 4
  • Automated caching of binaries, shared inputs, and task-specific data avoids direct GPFS use and brought execution efficiency within a few percent of ideal.The simple caching scheme supported scaling to 128K processors where the same workloads did not scale well beyond 8K; task-specific data is cached locally before persistence to the shared file system.Source: §III · p. 4

A. Swift and Falkon

Swift provides a data-flow workflow model, while Falkon dispatches tasks for large-scale applications. The Blue Gene/P implementation extends Falkon with alternate communication and distributed-dispatcher designs, addressing scalability, login-node load, and reliability.

A. Swift and Falkon

  • Swift integrates workflow scheduling and load balancing with Falkon’s lightweight task dispatcher, and the two systems have supported applications across clusters, Grids, and supercomputers.Reported applications span domains including molecular dynamics and economics, with runs reaching millions of tasks and hundreds of thousands of processors.Source: A. Swift and Falkon · p. 4

B. Implementation Details

  • Blue Gene/P support required engineering extensions, including a C executor and TCP-based communication to replace Java and Web Services components unavailable on its compute nodes.TCPCore manages dispatcher-side threads and persistent TCP sockets that can be reused across tasks.Source: B. Implementation Details · pp. 4–5
  • Distributing dispatchers across I/O nodes lets each manage 256 executors without inter-dispatcher communication, resolving the centralized design’s scaling problem and excess login-node load.The centralized design had difficulty scaling to 160K executors and 320K persistent sockets; administrators also objected to sustained login-node resource use.Source: B. Implementation Details · p. 5
  • Swift and Falkon handle failures at task granularity: failed-node tasks are affected, Falkon can reschedule known errors, and Swift restarts from persistent state without explicit checkpointing.I/O-node failures affect their respective 256-processor psets.Source: B. Implementation Details · p. 5

IV. MICROBENCHMARKS PERFORMANCE

Microbenchmarks characterize Blue Gene/P startup, dispatch, and task-efficiency costs. Falkon reaches thousands of tasks per second and scales to 160K processors, while efficiency at that scale depends on task length and dispatcher placement.

IV. MICROBENCHMARKS PERFORMANCE

  • The microbenchmarks measure startup, task dispatch, and shared-file-system operations; the Blue Gene/P experiments used a 160K-core system with 80 TB of memory.The system description reports a GPFS rating of 8 GB/s.Source: §IV · p. 5, A. System Descriptions · p. 5

B. Startup Costs

  • At 160K processors, preparing Falkon to process its first task takes 1,326 seconds, with Blue Gene/P booting accounting for 1,090 seconds.Mounting GPFS takes 708 seconds; boot-time operations account for over 97% of the 1,090-second boot.Source: B. Startup Costs · pp. 5–6, Figure 3 · p. 6

C. Falkon Task Dispatch Performance

  • 3,071 tasks/sec on Blue Gene/P with 640 dispatchers across 163,840 processors demonstrates high dispatch throughput at full scale.The same benchmarks report 1,758 tasks/sec with one dispatcher managing 4,096 processors.Source: C. Falkon Task Dispatch Performance · p. 6, Figure 4 · p. 6
  • In the Linux-cluster tests, Falkon reaches 2,534 tasks/sec versus 0.45 tasks/sec for PBS and 0.49 tasks/sec for Condor.The passage reports these rates for the same Linux environment.Source: C. Falkon Task Dispatch Performance · p. 6
  • For a 32K-task workload on 8K processors, Falkon completes in 30.31 seconds with 32 dispatchers, versus 182.85 seconds for Cobalt HTC-mode.For 1M tasks on 160K processors, Falkon’s reported overhead is 0.35 ms per task.Source: C. Falkon Task Dispatch Performance · p. 7
  • With one dispatcher, 4-second tasks achieve 95%+ efficiency through 2K processors; with distributed dispatchers, efficiency for 4-second tasks falls from 65% to 7% by 160K processors.At 160K processors, tasks of at least 64 seconds are needed for 90%+ efficiency; adding I/O increases the minimum task length.Source: C. Falkon Task Dispatch Performance · p. 7, Figure 5 · p. 7, Figure 6 · p. 7

D. Shared File System Performance

The section measures shared GPFS performance, which matters because many loosely coupled applications exchange files through the shared file system. Concurrent writes to one directory are costly at scale, while distributing them across directories improves performance.

D. Shared File System Performance

  • The experiments varied file sizes from 1KB to 10MB and processor counts from 4 to 16K, and tested both read-only and read+write throughput.For operation costs, they measured file and directory creation in one shared directory or across pre-created directories, as well as script invocation and a no-I/O task.Source: D. Shared File System Performance · pp. 7–8, Figure 8 · p. 8
  • At 16K processors, 10MB-file throughput reached 4.4GB/s for reads and 1.3GB/s for read+write operations.Measurements were made on a production system where more than 90% of the system was in use by other applications, which may have influenced results.Source: D. Shared File System Performance · p. 8, Figure 7 · p. 8
  • At 16K processors, creating a file or directory in the same directory took 404 or 1,217 seconds on average, respectively.The aggregate rates were 40 file creates and 13 directory creates per second; at 160K processors, creating that many files or directories would take 68 or 210 minutes.Source: D. Shared File System Performance · p. 8
  • Using a unique directory for each creation reduced the overhead to 11 seconds at 16K processors, leading the authors to recommend splitting writes across directories to avoid GPFS lock contention.The reported times assume all processors perform the operation concurrently; staggered task I/O can make per-operation times notably lower.Source: D. Shared File System Performance · p. 8

V. LOOSELY COUPLED APPLICATIONS

The section evaluates loosely coupled applications on Blue Gene/P, including DOCK molecular-docking workloads. The DOCK runs scaled to large core counts, with high sustained utilization and performance dependent on managing shared-file-system use.

V. LOOSELY COUPLED APPLICATIONS

  • Synthetic benchmarks do not always predict how real applications with real I/O will behave, motivating evaluation with applications from multiple domains.The authors identify economic energy modeling and molecular dynamics among the application domains studied.Source: §V · p. 8

A. Molecular Dynamics: DOCK

  • The DOCK workload screened 15,351 KEGG compounds and existing drugs against nine proteins using DOCK6 to compute binding affinities.The section also identifies DOCK and MARS as applications used to evaluate many-task computing on Blue Gene/P.Source: A. Molecular Dynamics: DOCK · p. 8, §V · p. 8
  • 138,159 DOCK6 runs took 2,807 seconds on 128K cores, sustaining 95% utilization for roughly 600 seconds but achieving 30% overall utilization.Task durations varied from 23 to 2,802 seconds on average and the authors overlapped another application, which used idle CPUs at 97% utilization.Source: A. Molecular Dynamics: DOCK · p. 9, Figure 9 · p. 9
  • 934,803 DOCK5 runs completed on 116K CPUs in 2.01 hours, with 99.6% sustained utilization and 78% overall utilization.Only 116K of 128K allocated CPUs registered successfully; the authors attributed this startup issue to GPFS contention and later avoided it by moving Falkon to RAM and pre-creating log directories.Source: A. Molecular Dynamics: DOCK · p. 9, Figure 10 · p. 9
  • 99.7% efficiency was achieved for DOCK when comparing the same workload at 64K and 128K CPUs, with scaling to 116K CPUs out of 160K.The authors report this result only after measures to avoid the shared file system, including caching multi-megabyte application binaries and 35MB of static input data.Source: A. Molecular Dynamics: DOCK · p. 9

B. Economic Modeling: MARS

The MARS economic model was run for large petroleum-refining parameter studies on Blue Gene/P, while Swift added workflow management around Falkon execution. Swift’s data and bookkeeping overheads required optimization to recover high efficiency at 8K processors.

B. Economic Modeling: MARS

  • MARS models more than 20 petroleum-refining processes and supports detailed multivariable studies of economic and environmental parameters.Its runs range from 0.5 seconds for one iteration to hours for many thousands of iterations.Source: B. Economic Modeling: MARS · p. 9
  • The MARS parameter sweep tested how diesel yields from two crude-oil grades affect investment needed to maintain petroleum production over four decades.The larger workload combined internal and external parameter sweeps.Source: B. Economic Modeling: MARS · pp. 9–10
  • 88% overall efficiency accompanied a speedup of 115,168X against an ideal 130,816X for the large MARS workload.The workload comprised 1M tasks on 128K processors, with eight processor cores per task.Source: B. Economic Modeling: MARS · pp. 9–10, Figure 11 · p. 10

C. Running applications through Swift

  • In the 8K-processor MARS run, shared-file-system load from other applications slowed execution compared with the 128K-processor run, where the experiment had dedicated access.The passage attributes the difference to file-system activity, including application loading and input/output.Source: C. Running applications through Swift · p. 10
  • Swift’s data staging, per-task directories, and status tracking reduced efficiency to 20% before shared-filesystem optimizations raised it to 88% on 8,192 processors.The optimizations used compute-node ramdisks for temporary directories, input data, and per-job logs.Source: C. Running applications through Swift · p. 10

D FUTURE WORK

The paper reports large-scale execution of loosely coupled applications on petascale systems, including Blue Gene/P runs up to 128K processors and microbenchmarks up to 160K. It also identifies system and application challenges that constrain efficient use.

D FUTURE WORK

  • The work demonstrated loosely coupled applications on Blue Gene/P at up to 128K processors and microbenchmarks at up to 160K processors.The authors present existing HPC systems as viable platforms for Many-Task Computing applications.Source: D FUTURE WORK · p. 11
  • The paper identifies resource-manager scalability and granularity, hardware utilization, shared-file-system contention, reliability, and application scalability as challenges.It also emphasizes understanding system limitations when identifying promising MTC applications.Source: D FUTURE WORK · p. 11
  • The paper contrasts tightly coupled MPI jobs with loosely coupled independent sequential jobs that can be scheduled individually and communicate through files.Blue Gene/P systems with up to 256K processors are described as available to the scientific community.Source: D FUTURE WORK · p. 11

A. Characterizing MTC Applications for Petascale Systems

The authors characterize suitable petascale MTC workloads by task count, task duration, and I/O volume per computation. Their Blue Gene/P experience identifies shared-file-system activity as a main bottleneck and motivates moving repeated operations into memory.

A. Characterizing MTC Applications for Petascale Systems

  • For 90%+ efficiency, the authors recommend average tasks longer than O(60 sec) with minimal I/O, or 1 second of compute per core per 5KB~50KB of I/O.These characteristics are based on Blue Gene/P experience at 160K CPU scale and a shared file system rated at 8GB/s.Source: A. Characterizing MTC Applications for… · p. 11
  • The shared file system was the main bottleneck; caching middleware, application binaries, and static input data in memory let repeated use avoid it.The authors report that startup costs can be amortized for a sufficiently large application.Source: A. Characterizing MTC Applications for… · p. 11
  • Concurrent small, line-buffered writes can severely slow or even crash the shared file system, so the authors recommend bulk reads and writes through memory.They also describe merging many output files into a single tar archive.Source: A. Characterizing MTC Applications for… · p. 11

B. Future Work

The future-work directions focus on reducing shared-file-system pressure through data management and collective I/O, while extending Falkon’s data-aware capabilities and support for parallel tasks. The proposed MPI support targets invocations smaller than the Blue Gene/P’s current processor-allocation granularity.

B. Future Work

  • They plan transparent data management that uses local file systems where possible to reduce reliance on shared storage.The motivation is that many MTC applications read and write substantial data.Source: B. Future Work · p. 11
  • The authors propose collective I/O over specialized interconnects to broadcast shared files or aggregate many node-local files before writing to GPFS.The goal is fewer file-system calls and less contention.Source: B. Future Work · p. 11
  • The authors plan to port data caching to Falkon’s C executor and use the Blue Gene/P Torus network for compute-node communication.They report having enabled TCP/IP connectivity over MPI on the Torus, providing a global IP space across compute nodes.Source: B. Future Work · p. 11
  • Planned Falkon support for MPI applications would allow each invocation to use an arbitrary processor count, including 32 processors rather than the current 256-processor pset granularity.The motivating use case involves thousands of separate MPI application invocations.Source: B. Future Work · p. 11
Loading 0808.3540v2…