Source-linked AI summary
DMTCP: Transparent Checkpointing for Cluster Computations and the Desktop
Jason Ansel, Kapil Arya, Gene Cooperman
TL;DR
DMTCP addresses the need for transparent checkpointing across desktop and distributed applications without system privileges or kernel changes. It checkpoints unmodified executables and coordinates complex process and socket state. On 128 distributed cores, checkpointing typically takes 2 seconds, or 0.2 seconds with forked checkpointing, with negligible runtime overhead and nearly constant timing as nodes increase; future work targets additional communication and debugging support.
Problem
Existing checkpointing had limited penetration into ordinary desktop applications and often depended on controlled environments, kernel modules, or application-specific restrictions.
Method
DMTCP uses transparent user-level checkpointing for unmodified Linux executables, coordinating distributed processes, threads, sockets, and restart metadata without kernel modification.
Results
2 seconds on 128 distributed cores, or 0.2 seconds with forked checkpointing, with negligible runtime overhead and nearly constant timing as nodes increase within a medium-size cluster.
Takeaways & Limitations
DMTCP supports broad scientific and desktop application coverage while enabling frequent checkpointing and minimal interruption for interactive applications.
Takeaways & Limitations
Future work includes support for multicast, RDMA, full ptrace support for gdb sessions, and expanded X-Windows checkpointing.
Abstract
from arXiv · showhide
DMTCP (Distributed MultiThreaded CheckPointing) is a transparent user-level checkpointing package for distributed applications. Checkpointing and restart is demonstrated for a wide range of over 20 well known applications, including MATLAB, Python, TightVNC, MPICH2, OpenMPI, and runCMS. RunCMS runs as a 680 MB image in memory that includes 540 dynamic libraries, and is used for the CMS experiment of the Large Hadron Collider at CERN. DMTCP transparently checkpoints general cluster computations consisting of many nodes, processes, and threads; as well as typical desktop applications. On 128 distributed cores (32 nodes), checkpoint and restart times are typically 2 seconds, with negligible run-time overhead. Typical checkpoint times are reduced to 0.2 seconds when using forked checkpointing. Experimental results show that checkpoint time remains nearly constant as the number of nodes increases on a medium-size cluster. DMTCP automatically accounts for fork, exec, ssh, mutexes/semaphores, TCP/IP sockets, UNIX domain sockets, pipes, ptys (pseudo-terminals), terminal modes, ownership of controlling terminals, signal handlers, open file descriptors, shared open file descriptors, I/O (including the readline library), shared memory (via mmap), parent-child process relationships, pid virtualization, and other operating system artifacts. By emphasizing an unprivileged, user-space approach, compatibility is maintained across Linux kernels from 2.6.9 through the current 2.6.28. Since DMTCP is unprivileged and does not require special kernel modules or kernel patches, DMTCP can be incorporated and distributed as a checkpoint-restart module within some larger package.
1 Introduction
DMTCP extends transparent checkpointing from controlled batch environments to unmodified desktop and distributed applications without system privileges. Its feature combination supports migration, debugging, workspace restoration, and other use cases beyond traditional batch checkpointing.
- DMTCP targets both typical desktop applications and distributed, multithreaded cluster computations through a user-level package requiring no system privileges.The package can be bundled with applications and supports transparent operation without recompilation or relinking.
- Use cases: DMTCP enables CPU-intensive computation to run on a powerful host or cluster before restarting all processes on a single laptop for interactive analysis.
- Use cases: Checkpointing lets programmers repeatedly restart long-running jobs from just before a bug, reducing the debug-recompile cycle.
- DMTCP combines user-level operation, multithreaded distributed processes, socket connectivity, fast checkpoints, and negligible non-checkpointing overhead.
- Use cases: DMTCP eliminates application-specific save/restore workspace commands and supports debugging distributed applications by checkpointing all processes before a bug.
- Use cases: DMTCP also supports long-running distributed applications, automatic rollback to safe mode after detected races or deadlocks, and checkpoint images as bug reports.
2 Related Work
Prior checkpointing systems span user-level MPI-specific tools, kernel-level approaches, and virtual machines, each with deployment or compatibility constraints. DMTCP instead pursues general distributed scientific software without kernel modification or dependence on a specific message-passing library.
- DejaVu offers related transparent socket-based checkpointing, but reported Chombo results show 45% overhead versus DMTCP's essentially zero overhead between checkpoints.
- User-level MPI checkpointing supports distributed processes only when they communicate exclusively through MPI and is typically restricted to one MPI dialect.
- Kernel-level checkpointing requires kernel modification and matching package and kernel versions, making portability and maintenance difficult.
- BLCR checkpoints processes on one machine, while selected MPI libraries integrate with it to provide distributed checkpointing.
- DMTCP aims to support more general distributed scientific software rather than relying on MPI-library communication hooks.
- Compared with virtual-machine snapshots, process-level DMTCP is lighter weight, easier to apply to distributed applications, and does not require installation privileges.
3 Usage and Features
DMTCP provides command-line checkpoint, control, and restart operations for recursively managed local and remote processes. Applications remain unaware of DMTCP by default, while an optional library exposes programmatic checkpoint controls.
- Users typically invoke dmtcp checkpoint to register a program, dmtcp command to control it, and dmtcp restart script.sh to restart the computation.
- Checkpoint registration recursively includes local and remote child processes, including MPI resource-management processes.
- DMTCP transparently intercepts ssh-spawned remote processes so they also run under DMTCP.
- Users can request checkpoints manually, periodically with --interval, or through the DMTCP programming interface.
- DMTCP writes per-process checkpoint images and generates a restart shell script containing restart commands for each node.
- DMTCP checkpoints unmodified Linux executables, while dmtcpaware.a can request or delay checkpoints, query status, and install checkpoint or restart hooks.
4 Software Architecture
DMTCP uses an injected user-level library with a two-layer architecture: DMTCP coordinates distributed artifacts while MTCP checkpoints individual processes. Its coordinated algorithms preserve shared resources across checkpoint and restart through barriers, socket handling, and process reconstruction.
- Two-layer architecture: DMTCP separates distributed checkpointing from single-process checkpointing, delegating individual-process state to MTCP through a small API.The separation supports maintenance, portability, and potential reuse over other single-process checkpointing packages.
- Transparent integration: Checkpointing is added transparently by injecting a shared library that launches a manager thread and wraps selected libc functions.The wrappers record socket, process-creation, and related metadata, while system calls and proc are used to inspect kernel state.
- Checkpoint protocol: All processes and threads are suspended together, while network data is moved from kernel buffers into user memory before the checkpoint image is written.Afterward, the data is retransmitted to refill socket buffers before application threads resume.
- Checkpoint protocol: The checkpoint protocol uses seven stages and six global barriers, with centralized barriers chosen for implementation simplicity.The algorithm executes asynchronously in each user process and uses a cluster-wide barrier as its communication primitive.
- Restart protocol: Restart reconstructs shared sockets by assigning globally unique connection IDs, restoring sockets in one process per host, then forking user processes.A discovery service re-establishes connections when processes move between hosts, after which file descriptors are rearranged to match the checkpointed layout.
- Restart protocol: The restart sequence first reopens files and recreates pseudo-terminals, then reconnects sockets before restoring the user-process descriptor arrangements.The unified restart process and subsequent fork are required to recreate resources shared between processes.
5 Experimental Results
DMTCP is evaluated across single-node desktop-style applications and distributed scientific programs, including socket-, MPICH2-, and OpenMPI-based workloads. The experiments examine broad application coverage, checkpoint/restart timing, storage configurations, scaling with nodes, and memory growth.
- Application coverage: The evaluation covers over 20 applications, including shell-like languages, MATLAB, Python, TightVNC, RunCMS, and distributed scientific programs.Tests include both common single-computer applications and distributed workloads using direct sockets, MPICH2, and OpenMPI.
- Application coverage: RunCMS checkpoints in 25.2 seconds and restarts in 18.4 seconds from a 680 MB configuration with 540 loaded dynamic libraries.The compressed checkpoint image is 225 MB.
- Distributed applications: Figure 4 compares checkpoint times, restart times, and checkpoint sizes for distributed applications with and without compression on 32 nodes.The suite includes direct-socket, MPICH2, and OpenMPI applications, with timings repeated 10 times and reported as means with standard-deviation error bars.
- Compression effects: Compression is generally slowest when the uncompressed checkpoint image is largest, but NAS/IS compresses quickly because its unwritten bucket regions are likely mostly zeroes.The zero-heavy regions compress both quickly and efficiently.
- Scalability: Checkpoint and restart times remain insensitive to the number of nodes in the ParGeant4 scaling tests.Checkpointing proceeds asynchronously on each node, and the single coordinator is not a bottleneck in these experiments.
6 Conclusions and Future Work
DMTCP demonstrates transparent distributed checkpointing without dependence on a specific message-passing library or kernel modification. On 128 cores, typical checkpoint time is 2 seconds, or 0.2 seconds with forked checkpointing, while future work targets additional communication and graphical-session support.
- Conclusions: DMTCP supports transparent distributed checkpointing across scientific and desktop applications without depending on a specific message-passing library or kernel modification.The approach combines user-level operation, multithreaded and distributed-process support, and low checkpoint overhead.
- Conclusions: 2 seconds is the typical checkpoint time on 128 distributed cores, reduced to 0.2 seconds with forked checkpointing and accompanied by negligible run-time overhead.The reported timings remain nearly constant as nodes are added within a medium-size cluster.
- Future work: Future work targets multicast and RDMA communication, full ptrace support for checkpointing gdb sessions, and broader X-Windows checkpointing beyond TightVNC.These extensions are presented as planned enhancements to user-level checkpointing.