Source-linked AI summary
JuPyLive: Seamless Migration of Jupyter Notebook Resources from Laptop to HPC
Sima Attar-Khorasani, Matthias Lieber, Siavash Ghiasvand
TL;DR
Resource-intensive workflows can outgrow local hardware, while manual migration and HPC batch scheduling make moving between local and remote resources difficult. JuPyLive integrates one-click, bidirectional notebook migration and HPC status information into Jupyter. It provides on-demand switching, though busy-resource waits can leave local edits unpreserved and large notebook contexts can slow migration.
Problem
Resource-intensive workflows may exceed local hardware, while manual migration and HPC batch scheduling make access to remote resources cumbersome.
Source: §1 · p. 2Method
JuPyLive uses notebook checkpointing, automated HPC allocation, and remote synchronization to provide one-click bidirectional migration within Jupyter.
Source: §1 · pp. 2–3, §3.2 · p. 6Results
JuPyLive enables on-demand switching between local workstations and HPC resources through a seamless migration mechanism.
Source: §1 · pp. 2–3, §4 · p. 8Takeaways & Limitations
JuPyLive bridges local interactive development and HPC resources, with status information to help users choose when to migrate.
Source: §3.3 · p. 7, §4 · p. 8When users select busy or semi-busy resources, local notebook changes made while waiting are ignored, and large internal contexts may slow migration.
Source: §3.5 · p. 8
Abstract
from arXiv · showhide
This work introduces JuPyLive, a migration mechanism that enables seamless transition of Jupyter notebooks between local resources of user's workstation and remote resources of high-performance computing~(HPC) environments, while preserving the user experience. JuPyLive eliminates the underlying complexities of migration process, enabling users to freely choose among available local and remote resources, directly within the familiar Jupyter notebook environment via a single click. JuPyLive leverages ElasticNotebook to manage in-memory state migration, it automates resource allocation on HPC cluster and orchestrates required remote communication channels between the source and destination to enable a bidirectional migration. Furthermore, HPC status monitor of JuPyLive provides a live overview of available remote resources, allowing users to make informed decisions on choosing the relevant resources before initiating a migration process. The proposed fully automatic mechanism requires no code changes or configurations by the end user, nor does it demand users to learn a new syntax, instead the migration process can be intuitively initiated and monitored using visual elements from within the Jupyter notebook. By bridging the gap between local workspace and remote resources, JuPyLive offers a seamless experience for scaling local resource-intensive workflows with minimal user intervention, thus further democratizing the usage of HPC clusters among the interdisciplinary researchers.
1 Introduction
Resource-intensive workflows create a need to move between local computers and HPC, but manual migration and HPC batch scheduling make that switch difficult. JuPyLive proposes a one-click migration mechanism within Jupyter notebooks.
1 Introduction
- Resource-intensive machine-learning workflows can exceed local hardware, while HPC access is costly and regulated.This creates demand for switching between resources even during early prototype development.Source: §1 · p. 2
- Manual migration of code and development environments imposes substantial overhead, while batch scheduling adds another barrier to interactive HPC use.The paper identifies Slurm as a common HPC scheduler.Source: §1 · p. 2
- JuPyLive proposes transparent, on-demand migration between local and HPC resources, emulating an elastic resource pool that adapts to user demand.The mechanism is intended to let users switch within Jupyter notebooks with a single click.Source: §1 · pp. 2–3
- The work builds on JupyterHub as a development environment, while noting that the mechanism could also be implemented in other environments..Source: §1 · p. 2
2 Related Work
Prior notebook-migration approaches address process- or application-level migration, but face constraints such as GPU support, environment compatibility, or manual setup. JuPyLive uses ElasticNotebook mechanisms and integrates Slurm scheduling into migration.
2 Related Work
- OS-level migration tools have limitations including GPU-workload support, platform dependence, and heterogeneous-environment migration [1].They also require manual configuration and expertise.Source: §2 · p. 3
- ELSA migrates CPU-based Jupyter notebooks across machines using CRIU and shared NFS, but CRIU’s lack of GPU support limits its use for GPU-dependent workflows [4].Source: §2 · p. 3
- Selective cell migration uses AST-based pruning for hardware-intensive cells, but assumes matching libraries and versions across source and destination [2].Source: §2 · p. 3
- ElasticNotebook chooses between transferring state and recomputing cells, and JuPyLive uses its mechanisms for notebook migration [5].The related-work passage identifies integration with batch schedulers such as Slurm as a gap.Source: §2 · p. 4
3 System Overview and Implementation
JuPyLive combines notebook-based controls, checkpoint and environment synchronization, HPC resource allocation, and status monitoring to support migration between local and HPC resources. Its current implementation has predefined resource requests and other stated operational constraints.
3 System Overview and Implementation
- JuPyLive uses ElasticNotebook checkpoint/restore and automates bidirectional migration through JupyterHub toolbar controls.Its workflow supports movement from local resources to HPC and back when local resources suffice.Source: §3 · pp. 4–5, Figure 1 · p. 4
3.1 Frontend Integration
- Toolbar buttons invoke migration and HPC-status functions through JupyterLab’s active kernel, handling partition input and redirecting the browser when the remote server is ready.Source: §3.1 · p. 5
3.2 Migration Engine
- Migration checkpoints notebook state, synchronizes project files, aligns remote dependencies, and requests HPC resources with hardware-specific flags such as GPU requirements.The remote bootstrap restores the session configuration, starts a Jupyter server, and reconnects the browser through a reverse SSH tunnel.Source: §3.2 · pp. 5–6
- A local Pull Watcher detects a remote return-migration signal, pulls updated files and the checkpoint, and restores the notebook locally.The migration can be repeated as needed.Source: §3.2 · p. 6
3.3 HPC Telemetry and Decision Support
- HPC Status concurrently gathers partition resources, availability, and queue lengths, then presents color-coded readiness recommendations for connected clusters.Figure 2 depicts HPC status, queue depths, and availability for each cluster.Source: §3.3 · p. 7, Figure 2 · p. 7
3.4 Automatic Environment Preparation
- A zero-configuration installer prepares system dependencies, a Python environment, SSH access, and the local Jupyter setup.It enables users to start coding in Jupyter without manual configuration steps.Source: §3.4 · pp. 7–8
3.5 Limitation and Assumptions
- Requested HPC resource amounts are predefined, cluster access points are configured manually, and non-Slurm platforms require administrator adaptation.Busy or semi-busy allocations may take time; local edits made while waiting are ignored, and large notebook contexts may slow migration. The remote resource must be accessible from the local machine.Source: §3.5 · p. 8
4 Conclusion and Future Direction
JuPyLive demonstrates a mechanism for on-demand migration between local workstations and HPC, automating state, file, and network orchestration. Future work targets synchronization, resource allocation, and migration recommendations.
4 Conclusion and Future Direction
- JuPyLive bridges interactive development on lower-performance local resources with batch-processing HPC resources by orchestrating state, files, and network routing.The current implementation demonstrates the feasibility of one-click migration across heterogeneous environments.Source: §4 · p. 8
- Planned work includes optimizing synchronization, using shared storage to reduce transfers, dynamically allocating resources, and recommending migration targets from historical and live data.The planned recommendations would use migration patterns, queue conditions, and resource utilization records.Source: §4 · p. 9