Source-linked AI summary
Context-aware Execution Migration Tool for Data Science Jupyter Notebooks on Hybrid Clouds
Renato L. F. Cunha, Lucas V. Real, Renan Souza, Bruno Silva, Marco A. S. Netto
TL;DR
Interactive notebooks can benefit from execution across local and remote platforms, but deciding which cells to migrate, when, and how is challenging. The paper presents a Jupyter extension that combines interaction-aware migration decisions with notebook-state reduction. Block-cell migration outperforms single-cell migration across the tested combinations of migration times and full remote speedups.
Problem
Choosing which notebook cells to migrate, when, and how requires weighing factors such as execution time, cost, privacy, data location, and available resources.
Source: §I · p. 1Method
A Jupyter extension combines an interaction-pattern-aware cell migration policy with AST- and dependency-based notebook-state reduction.
Source: §I · p. 1Results
Block-cell migration outperforms single-cell migration across all tested combinations of full remote speedups and migration times.
Source: C. Result Analysis · p. 7Takeaways & Limitations
Considering common user cell sequences can avoid unnecessary migrations compared with decisions based only on the active cell's computational needs.
Source: §V · p. 10
Abstract
from arXiv · showhide
Interactive computing notebooks, such as Jupyter notebooks, have become a popular tool for developing and improving data-driven models. Such notebooks tend to be executed either in the user's own machine or in a cloud environment, having drawbacks and benefits in both approaches. This paper presents a solution developed as a Jupyter extension that automatically selects which cells, as well as in which scenarios, such cells should be migrated to a more suitable platform for execution. We describe how we reduce the execution state of the notebook to decrease migration time and we explore the knowledge of user interactivity patterns with the notebook to determine which blocks of cells should be migrated. Using notebooks from Earth science (remote sensing), image recognition, and hand written digit identification (machine learning), our experiments show notebook state reductions of up to 55x and migration decisions leading to performance gains of up to 3.25x when the user interactivity with the notebook is taken into consideration.
I. INTRODUCTION
Interactive notebooks can benefit from using multiple computing platforms, but deciding which cells to move, when, and how is challenging. The paper introduces an extension, interaction-aware migration policy, and notebook-state reduction, then evaluates them on several notebook types.
I. INTRODUCTION
- The migration challenge is deciding which cells to move, when, and how while balancing execution time, cost, privacy, data location, access restrictions, and specialized resources.Notebook workloads may benefit from splitting processing across platforms, such as remote data processing with visualization near the user.Source: §I · p. 1
- The Jupyter extension automatically manages notebook-state migration without requiring users to modify their source code.The system also annotates migration decisions with explanations, including performance gains for remote execution.Source: §I · p. 1, Figure 1 · p. 2
- The migration policy uses user-interaction patterns alongside cell content, migration costs, and target-platform performance to decide when and how to migrate cells.Source: §I · p. 1
- The notebook-state reducer tracks code-cell dependencies and uses abstract syntax trees to reduce state size, decreasing migration time.Experiments use notebooks from Earth science, image recognition, and handwritten-digit identification.Source: §I · p. 1
II. SOLUTION
The proposed solution helps users run interactive notebooks across multiple computing platforms; the following sections describe it.
II. SOLUTION
- The solution is intended to let users utilize multiple computing platforms when using interactive computing notebooks.Source: §II · p. 2
A. Extending JupyterLab for Context-aware Migration
The extension operates in JupyterLab, where it monitors user actions and transparently adapts code-cell execution across platforms. It uses front-end event hooks, telemetry, and an injected kernel preamble to support context detection and cell migration.
A. Extending JupyterLab for Context-aware Migration
- The paper focuses on JupyterLab, a web-based IDE, and its extension operates on code cells while ignoring other cell types.Jupyter notebook files are represented as lists of code, attachment, or raw cells.Source: A. Extending JupyterLab for… · p. 2
- Although implemented in the back end, the extension needs awareness of user actions to detect context and decide when to migrate cells.Source: A. Extending JupyterLab for… · p. 2
- The extension forwards telemetry from front-end actions through an authenticated endpoint to a message queue, with messages carrying session, notebook, cell, path, and event information.The extension seeks to avoid adding networking requirements on top of JupyterLab.Source: A. Extending JupyterLab for… · p. 2
- Event listeners and a runtime patch to JupyterLab's kernel-execution function let the extension analyze interactions and transparently modify user code.Source: A. Extending JupyterLab for… · p. 3
- At session start or kernel restart, an injected preamble connects to execution engines and registers a cell magic that finds dependencies and can compress data before migration.Source: A. Extending JupyterLab for… · p. 3
B. Context Detector
The context detector analyzes users’ notebook interaction histories to identify recurring sequences of executed or modified cells. It scores these sequences by frequency and returns sequences containing the currently active cell.
B. Context Detector
- The detector uses notebook interaction history and current cell order to return scored sequences containing the active cell.Its inputs are the current cell order and historical cell orders.Source: B. Context Detector · p. 3
- A sequence is a nondecreasing list of cell positions, split whenever the next position creates a gap.For example, 1, 2, 3, 2, 3 contains the sequences 1, 2, 3 and 2, 3.Source: B. Context Detector · p. 3
- Duplicate and subsequence sequences are counted and normalized into percentage scores reflecting their frequency in the interaction history.The algorithm removes duplicate entries while accounting for their counts, then normalizes the scores.Source: B. Context Detector · p. 3
C. Context-aware Migration Analyzer
The migration analyzer combines knowledge about cell contents with execution-time estimates to decide whether cells or blocks should run remotely. It updates parameter estimates in the background and uses notebook interaction patterns to guide block migration.
C. Context-aware Migration Analyzer
- The analyzer applies knowledge-aware and performance-aware policies, using cell contents or prior execution behavior to decide whether to migrate.The context detector can identify groups of cells for block migration.Source: C. Context-aware Migration Analyzer · p. 3
- The knowledge-aware policy compares cell parameters, such as epochs or batch size, with stored estimates of when remote execution is worthwhile.An expert may provide initial estimates, which are dynamically updated as users write and execute cells.Source: C. Context-aware Migration Analyzer · p. 4
- For performance-aware migration, remote execution is selected when predicted remote time is less than local execution time plus migration time.The policy uses estimates of local execution time, migration time, and remote speedup.Source: C. Context-aware Migration Analyzer · p. 4
- Single-cell migration transfers notebook state to the remote host and back for each cell, while block migration uses interaction patterns to migrate only when needed.A predicted group runs remotely until its cells are completed or the user runs a cell outside the predicted sequence.Source: C. Context-aware Migration Analyzer · p. 4, Figure 3 · p. 4
- For model training, the fitting cell is treated as a cell of interest, and background local executions with fast parameter values build its training dataset.Cell events also prompt background updates that inspect notebook cells for parameters known to the knowledge base.Source: C. Context-aware Migration Analyzer · p. 4
- The analyzer measures short executions locally and remotely, then trains separate models to predict execution times for longer-running parameter values.Initial measurements use varying small epoch counts; the dataset is updated after relevant notebook changes or cell executions, and prediction intersections update the knowledge base.Source: C. Context-aware Migration Analyzer · pp. 4–5
D. Notebook State Reducer
Interactive notebook state can accumulate unused objects, increasing serialization and transfer time during migration. The reducer analyzes dependencies for the selected cell and temporarily detaches unneeded objects before serialization.
D. Notebook State Reducer
- Interactive execution can leave a large runtime memory footprint, making notebook-state serialization and network transfer slower during migration.The delay can be compounded when state must be transferred back to the original host.Source: D. Notebook State Reducer · p. 5
- The reducer analyzes a cell’s source with an abstract syntax tree and recursively identifies needed variables, functions, their dependencies, and loaded modules.Runtime dependency analysis avoids processing branches not taken in the current execution and captures changes influenced by user input.Source: D. Notebook State Reducer · p. 5
- Objects outside the dependency list are temporarily detached so serialization transfers the state needed for remote execution, then they are reattached afterward.If serialization fails, the cell executes locally.Source: D. Notebook State Reducer · p. 5
- After the first local-to-remote migration, later transfers serialize only changed objects, with optional compression to reduce network transfer time.Unhasheable objects are always migrated, potentially transferring more data than needed.Source: D. Notebook State Reducer · p. 5
III. EVALUATION
The evaluation has two experiment sets: one studies notebook-state migration, and the other examines interactive execution under different cell-migration strategies.
III. EVALUATION
- The evaluation comprises a notebook-state migration study and an interactive-execution study comparing migration strategies.Source: §III · pp. 5–6
- The notebook-state experiments examine migration memory consumption and time for selected notebook cells.Source: §III · pp. 5–6
- The interactive-execution experiments compare strategies ranging from local to remote execution.Source: §III · p. 6
A. Notebook State Reducer for Migration
The notebook-state reducer is evaluated on a satellite-image processing notebook, with state-size reductions reported for both outbound and return migrations.
A. Notebook State Reducer for Migration
- The tested notebook processes 720 satellite images from 30 regions, filters similar scenes, and retains 93 distinct images for further processing.Source: A. Notebook State Reducer for Migration · p. 6
- 55× with compression and 8× without: the reducer shrinks the original notebook state before migrating the selected compute-intensive image-processing cell.The cell applies a Sobel filter, then K-Means clustering and shapefile conversion.Source: A. Notebook State Reducer for Migration · p. 6
- 13× compressed and 5× uncompressed: the approach reduces return-transfer data size versus the full notebook state.The reducer identifies new and changed objects in the remote state and transfers only those back.Source: A. Notebook State Reducer for Migration · p. 6
- Figure 4 presents user interaction for the synthetic-loops and adapted TensorFlow notebooks, showing user execution index and current executed-cell order.Source: Figure 4 · p. 6
B. Context-aware Migration
The context-aware migration evaluation compares policies on two notebooks with recorded interaction patterns, using specified local and remote computing environments.
B. Context-aware Migration
- Four policies are compared: local execution, single-cell migration, context-informed block-cell migration, and remote execution.The experiments impose fixed migration time and remote speedups, and assess performance- and knowledge-aware variations for single-cell and block-cell migration.Source: B. Context-aware Migration · p. 6
- The two test notebooks are synthetic loops and an adapted TensorFlow MNIST tutorial; recorded user cycles, including repeated execution of cells 1–7, inform migration decisions.Source: B. Context-aware Migration · p. 6
- The local system has 6 physical CPU cores and 16 GB RAM, while the remote cluster node has 56 CPU cores, 1.5 TB RAM, and an NVIDIA K80 GPU.The software stack includes TensorFlow 1.15 and Keras 2.3.1 for the training notebook.Source: B. Context-aware Migration · pp. 6–7
C. Result Analysis
The analysis compares single-cell and block-cell migration policies with full local and remote execution, and evaluates performance- and knowledge-aware policies. Block-cell migration outperforms single-cell migration across tested settings; the size of its advantage depends on notebook execution patterns and migration conditions.
Performance-aware policy
- Block-cell speedups are higher for synthetic loops than for the TensorFlow guide, where shorter execution cycles and more lightweight cells increase migration frequency.The synthetic loops notebook has more scattered execution times, while the TensorFlow guide has two main groups of cell execution times.Source: C. Result Analysis · p. 7, Figure 7 · p. 8
- The block-cell/single-cell speedup ratio rises with remote speedup, but migration-time increases affect the two notebooks differently as migration counts change.For the TensorFlow guide, block-cell migration performs better at higher migration times; for synthetic loops, changing cell execution times can alter migration counts and the ratio.Source: C. Result Analysis · pp. 7–8, Figure 10 · p. 8, Figure 8 · p. 8
- Block-cell migration outperforms single-cell migration across tested migration-time and remote-speedup combinations, with the largest gains at low migration times and high remote speedups.The TensorFlow guide shows less significant block-cell speedups than the synthetic loops notebook because its shorter execution blocks lead to more migrations.Source: Figure 5 · p. 7, C. Result Analysis · p. 7, Figure 6 · p. 7
Knowledge-aware policy
- After epoch 7, migration pays off for training on a remote environment that runs 4.43x faster than local execution.Linear regressors trained on three samples predict total times; the optimal epoch threshold is updated in the knowledge base for migration decisions.Source: C. Result Analysis · pp. 8–9, Figure 11 · p. 9
IV. RELATED WORK
Related work covers Jupyter session migration and deployment, notebook provenance and user-behavior studies, and process migration across platforms. The paper distinguishes its approach by using ASTs to identify notebook data dependencies for automatic migration without user code modifications.
Jupyter-based technologies
- JupyterHub checkpointing and migration targets idle-session resource use and cheaper resources, while XSEDE work focuses on scalable notebook deployment, including HPC jobs and container resilience.The JupyterHub approach can restore a complete session on the same or another resource.Source: §IV · p. 9
Provenance and user behavior
- Notebook provenance tools capture execution history or support experiment comparison, while related studies examine student engagement and how data scientists track experiments.The cited user-behavior study relies on interviews rather than tooling to collect behavior data.Source: §IV · p. 9
Process migration
- Process-migration research profiles applications and analyzes file access and variable liveness; this paper instead uses ASTs for notebook data dependencies and migration without user code changes.The approach is initiated through a JupyterLab extension that observes user interaction with notebooks.Source: §IV · p. 10
V. CONCLUSION
The paper presents a Jupyter notebook migration tool and a notebook-state reduction component. Its conclusions emphasize user interaction patterns as a basis for migration decisions and state reduction as a context-aware garbage collector.
V. CONCLUSION
- The proposed tool migrates notebook code to suitable computing platforms without requiring source-code changes, using a JupyterLab extension.It can execute notebook cells remotely and annotate cells to explain migration decisions.Source: §V · p. 10
- AST-based notebook-state reduction acts as a context-aware garbage collector without compromising creation of new cells.Interactive notebooks cannot predict which variables future cells may access, so memory consumption tends to grow over time.Source: §V · p. 10
- Migration decisions should use common sequences of executed cells to balance computational power and monetary cost and avoid unnecessary migrations.The conclusion contrasts user-context-aware decisions with decisions based only on the computational needs of the active cell.Source: §V · p. 10