Source-linked AI summary
Ganga: a tool for computational-task management and easy access to Grid resources
J. T. Mościcki, F. Brochu, J. Ebke, U. Egede, J. Elmsheuser, K. Harrison, R. W. L. Jones, H. C. Lee, D. Liko, A. Maier, A. Muraru, G. N. Patrick, K. Pajchel, W. Reece, B. H. Samset, M. W. Slater, A. Soroko, C. L. Tan, D. C. Vanderster, M. Williams
TL;DR
Researchers must work across distributed systems with different interfaces, and the effort to manage them can be prohibitive without appropriate tools. Ganga provides a common task-management environment with multiple interfaces and extensible plugins. Its use across many sites and users, and examples spanning scientific areas, demonstrate its deployment and adaptability.
Problem
Researchers increasingly face multiple distributed systems with different interfaces, and managing tasks across them can be prohibitive without appropriate tools.
Source: Introduction · p. 2Method
Ganga provides a common job-management interface with GUI, Python command-line, and scripting access, while plugins connect applications and backends.
Source: Functionality · p. 4, §7.3 · p. 21Results
From January 2007 to December 2008, Ganga was used at 150 sites by 2,000 unique users running about 250,000 sessions.
Source: Introduction · p. 4Takeaways & Limitations
Ganga supports local development followed by Grid execution and can be extended for new user communities and scientific areas.
Source: Conclusion · p. 23
Abstract
from arXiv · showhide
In this paper, we present the computational task-management tool Ganga, which allows for the specification, submission, bookkeeping and post-processing of computational tasks on a wide set of distributed resources. Ganga has been developed to solve a problem increasingly common in scientific projects, which is that researchers must regularly switch between different processing systems, each with its own command set, to complete their computational tasks. Ganga provides a homogeneous environment for processing data on heterogeneous resources. We give examples from High Energy Physics, demonstrating how an analysis can be developed on a local system and then transparently moved to a Grid system for processing of all available data. Ganga has an API that can be used via an interactive interface, in scripts, or through a GUI. Specific knowledge about types of tasks or computational resources is provided at run-time through a plugin system, making new developments easy to integrate. We give an overview of the Ganga architecture, give examples of current use, and demonstrate how Ganga can be used in many different areas of science.
Introduction
Researchers face an expanding range of distributed systems with different interfaces, making task configuration, tracking, and result management burdensome without suitable tools. Ganga provides a common task-management interface that supports development locally and execution on Grid resources.
Introduction
- Researchers may need substantial effort to learn unfamiliar distributed systems and manage task configuration, tracking, archives, and results.The paper describes distributed systems as increasingly common and says the burden can be prohibitive without appropriate tools.Source: Introduction · p. 2
- Ganga provides an easy-to-use frontend for configuring, executing, and managing computational tasks, initially developed for ATLAS and LHCb data analysis.It was designed for large communities needing Grid resources for data mining and simulation.Source: Introduction · p. 2
- Ganga’s interface supports interactive Python use, a GUI, and scripts, while its plugin system allows expert customization of applications and resources at runtime.Custom application classes can encapsulate repeated low-level setup and hide differences such as local versus Grid data access.Source: Introduction · pp. 2–3
- Ganga lets users move jobs from local execution to Grid processing by changing one job parameter, supporting progression from prototyping to large-data analysis.The paper also describes using local applications for quick development turnaround.Source: Introduction · pp. 2–3
- Ganga can also act as a library for job submission and control, including as a building block for domain-specific Grid portals.The paper states that Ganga generally does not install Grid or batch submission tools or application software; these are typically managed separately by system administrators.Source: Introduction · p. 3
- Between January 2007 and December 2008, Ganga was used at 150 sites around the world, with 2,000 unique users running about 250,000 Ganga sessions 4.Source: Introduction · p. 4
Functionality
Ganga represents computational work as jobs assembled from components and exposes them through interfaces suited to different working styles. Its components cover application setup, execution backends, datasets, parallel subjobs, and output merging.
Functionality
- Users access Ganga through Python command-line interaction, scripts, or a GUI, and jobs combine application and backend components with optional data, splitting, and merging components.The application and backend are required; other components are optional.Source: Functionality · p. 4, Figure 2 · p. 6
2122.4 Splitter components
- Splitter components define how subjobs differ, enabling independent workloads to run separately and their outputs to be merged after completion.An ArgSplitter varies application arguments; specialized splitters can divide processing across different parts of a task.Source: Functionality · p. 5, 2122.4 Splitter components · p. 9
- Ganga can manage subjob collections for more sophisticated parallelization schemes, including MPI processes.Some backends optimize bulk submission, and Ganga can use a backend’s job-collection mechanism transparently.Source: Functionality · pp. 5–6
2.1 Application components
- Application components define software settings and actions before and after processing, including configuration checks and output validation.The basic Executable component includes an executable path, arguments, and environment variables.Source: §2.1 · p. 7
2.2 Backend components
- Backend components describe processing-system behavior and provide methods to submit, cancel, monitor, and retrieve jobs across supported local, batch, and Grid systems.Backend parameters can include queues and resource requirements; backends can also submit to systems accessible through a remote machine.Source: §2.2 · pp. 7–8
2.3 Dataset components
- Dataset components identify data collections and provide information such as location and size, while sandbox files transferred with a job are intended for small files, typically up to 10 MB.The paper distinguishes externally stored datasets, which may be arbitrarily large, from job sandbox files.Source: §2.3 · pp. 8–9
2212.5 Merger components
- Merger components combine subjob outputs, such as concatenating standard output and error or summing ROOT histograms.Merging can occur automatically when output is retrieved or be controlled manually.Source: 2212.5 Merger components · p. 9
2303 Implementation
Ganga uses plugins to connect user-facing job components to underlying execution technologies and preserves job records and files through repositories and workspaces. These abstractions support extensibility and access from multiple locations.
3.1 Components
- Ganga’s plugin architecture lets component classes implement abstract interfaces, with schemas defining what users see through generated proxy classes.A submission handler translates an application’s internal representation into a form accepted by a specific backend, without requiring every application-backend combination to be supported.Source: Figure 3 · p. 10, §3.1 · pp. 10–11
3.2 Job persistence
- The job repository automatically saves jobs and their metadata for later sessions and supports searching and bookkeeping.Local and AMGA-backed remote repositories scale to 10,000 jobs per user in tests, with average individual job creation taking about 0.2 seconds.Source: §3.2 · p. 11
3.3 Input and output files
- A workspace stores each job’s input and output files, while an abstraction layer allows access through Ganga commands and integration of additional workspace implementations.A WebDav prototype allowed job workspace data to be accessed from different locations, with a local workspace cache remaining available.Source: §3.3 · p. 12
- Combining a remote workspace with a remote repository creates a roaming profile that makes the same Ganga session accessible from multiple locations.The paper compares this access pattern to reading email from an IMAP server.Source: §3.3 · p. 12
Monitoring
Ganga monitors job progress internally while supporting external services that report additional execution metrics. Its internal polling and external sensors cover jobs from submission through completion.
4.1 Internal monitoring
- Backend polling runs in separate pooled threads, accommodates varying backend response times and load, and allows a separately configured poll rate for each backend.An efficient mechanism avoids deadlocks when backends respond slowly.Source: §4.1 · p. 12, Figure 4 · p. 13
- The monitoring subsystem tracks credential validity and notifies users when renewal is required; without renewal, operations needing valid credentials are disabled.Source: §4.1 · p. 13
4.2 External Monitoring
- External sensors can be added dynamically on client and worker nodes, with events generated at submission, startup, periodically during execution, and at completion.Sensors can report different job metrics; current implementations include ATLAS Dashboard monitoring and an optional real-time output service configured for each user community.Source: §4.2 · pp. 13–14
5 Graphical User Interface
The GUI exposes Ganga's job-management functionality through a multithreaded interface, with an overview of jobs and details for an individual job. It also supports job construction, monitoring, and script-based work.
5 Graphical User Interface
- The multithreaded GUI runs monitoring and most job-management actions concurrently, providing a degree of parallelism unavailable at the command line.Source: §5 · p. 14
- The interface places the job overview on the left and individual job details on the right.Source: Figure 5 · p. 14
- A job-builder window supports creating and editing jobs with a foldable attribute tree and data-entry widgets generated from plugin schemas.Drop-down menus and tooltips assist data entry, while buttons provide job actions including saving, submission, termination, and removal.Source: §5 · p. 15
- The scriptor window combines a favourite-scripts collection, job-script editor, and embedded Python session for working with Ganga scripts and commands.Source: §5 · p. 15
6 Use in experiments at the Large Hadron Collider
Ganga serves ATLAS and LHCb analysis users working across distributed computing resources, including several Grid infrastructures. The examples describe application-specific configuration and job management for both experiments.
6 Use in experiments at the Large Hadron Collider
- Ganga acts as the analysis interface for many ATLAS and LHCb physicists and allows users to exchange jobs despite their complex configurations.Source: §6 · p. 16
6.1 The LHCb experiment
- For Gaudi-based LHCb jobs, Ganga prepares the application environment and required libraries; users specify the application name and version and a configuration file.The application component parses that file and registers the required inputs and expected outputs with the submission backend.Source: §6.1 · p. 16
- LHCb splitting queries a file catalogue to place subjobs where all their data are available, avoiding pre-analysis transfers across the network.Source: §6.1 · p. 17
- Above 300k user jobs finished successfully in 2008, consuming 87 CPU years across 140 Grid sites, with spikes of several thousand simultaneous jobs.Source: §6.1 · p. 17
6.2 The ATLAS experiment
- ATLAS analysis scenarios requiring fast response and high interaction are suited to PROOF, while lower-interaction analyses with longer acceptable response times are suited to Ganga and Grid processing.Source: §6.2 · p. 18
6.2.1 ATLAS Grid infrastructures
- Ganga submits jobs across the three ATLAS Grid infrastructures: EGEE through gLite, OSG through PanDA, and NorduGrid through ARC.Source: §6.2.1 · p. 18
6.2.2 ATLAS user analysis
- ATLAS user-analysis jobs query DQ2 for dataset locations, restrict possible Grid sites accordingly, and can split processing across subjobs.After Athena completes, outputs are stored at the running site's storage element and registered in DQ2.Source: §6.2.2 · pp. 18–19
6.2.3 ATLAS small-scale event simulations
- The AthenaMC component handles input parameters and datasets across ATLAS event generation, detector simulation, and event reconstruction.Source: §6.2.3 · p. 19
5167 Other usage areas
Ganga’s extensible interface has supported use beyond particle physics, including industrial-scale image retrieval and integration with additional Grid middleware.
7.1 Enabling industrial-scale image retrieval
- Ganga enabled deployment of Imense’s image-analysis technology on the Grid to build a searchable index of more than twenty-million high-resolution photographs.The processing stages are sequential, but analysis runs independently per image or small subset, with a few hundred images grouped per job to reduce overhead.Source: §7.1 · p. 20
- Specialised Ganga application and dataset components let Imense jobs run locally or on the Grid, while returning results to the requested output location.The dataset component automatically retrieves and unpacks completed job archives, so the user-facing result is independent of the backend.Source: §7.1 · pp. 20–21
7.2 Smaller collaborations in High Energy Physics
- A small gaseous-detector community integrated Garfield Grid simulations using Ganga’s generic Executable component and a script in just a few hours.The jobs placed executables and small input files in the input sandbox and returned histograms and text output in the output sandbox.Source: §7.2 · p. 21
7.3 Ganga integrated with lightweight Grid middleware
- Ganga’s open-plugin architecture enabled ARC middleware integration through a backend that translates Ganga input into ARC-readable xRSL.The integration’s main user is ATLAS, where it provides access to one of the experiment’s three main computing grids.Source: §7.3 · pp. 21–22
8 Interfacing to other frameworks
Ganga’s public API can connect it to other frameworks, including DIANE, while its embedded use in a web portal offers a common interface for heterogeneous resources.
8 Interfacing to other frameworks
- Ganga’s Public Interface provides an API for generic job submission and management, allowing programmatic integration with other frameworks.In the DIANE integration, worker agents run as Ganga jobs, combining DIANE overlay scheduling with Ganga job management and a uniform interface for Grid and non-Grid resources.Source: §8 · p. 22
- The DIANE/Ganga approach has been used in medical physics, detector-simulation testing, molecular docking, telecommunications, and theoretical physics.The paper also reports low-latency Grid access and improved responsiveness for on-demand computing and interactivity.Source: §8 · p. 22
- An ASGC bioinformatics portal delegates job management to embedded DIANE/Ganga, letting users switch resources through one web interface.The portal is customized for analysis of candidate drugs against avian flu.Source: §8 · p. 22, Figure 7 · p. 23
Conclusion
The conclusion presents Ganga as a job-management tool for heterogeneous resources, supporting local task debugging followed by Grid execution and extension to new communities.
Conclusion
- Ganga supports defining tasks for local debugging and then running them on the Grid for large-scale data mining, with submission, monitoring, output retrieval, and bookkeeping.The paper describes this as simplifying task specification and job management in heterogeneous environments.Source: Conclusion · p. 23
- A Python prompt, GUI, and scripts provide alternative interfaces, while plugins allow Ganga to be extended and customised for new user communities.Examples cited include particle physics, medical physics, and image processing.Source: Conclusion · p. 23
- Compared with tools such as GridWay [53] and Condor-script interfaces, Ganga offers three working interfaces and a higher-level abstraction for domain-specific work.The conclusion says this lets users avoid changing their way of working when switching processing systems.Source: Conclusion · pp. 23–24
- The paper reports that Ganga has a large user base, is actively developed, and may support new scientific or commercial projects across distributed infrastructures.Source: Conclusion · p. 24
A Examples
The examples demonstrate creating and submitting jobs, moving a job from local execution to the Grid, and managing collections of jobs and subjobs.
A Examples
- A basic session creates and submits a job, then copies it, changes its backend to LCG, and submits the copy to the Grid.The example also shows inspecting output and listing jobs.Source: A Examples · pp. 24–25
- An LHCb analysis example uses a splitter to divide work into subjobs, with DIRAC routing them to locations holding the required data.The example assigns data by logical identifiers and shows querying subjob status and execution locations.Source: A Examples · p. 25
- Ganga’s job collections support selecting groups by status or name and resubmitting, killing, or submitting selected jobs.Source: A Examples · p. 25