Source-linked AI summary
ImJoy: an open-source computational platform for the deep learning era
Wei Ouyang, Florian Mueller, Martin Hjelmare, Emma Lundberg, Christophe Zimmer
TL;DR
Biomedical researchers face hurdles reusing deep learning tools, including difficult software setup and privacy concerns around uploading data. ImJoy is a browser-based platform with plugins for interactive image analysis and model use or retraining; its examples span browser-based phone analysis, large-image exploration, and nuclei segmentation.
Problem
Researchers face complex environment setup, limited programming accessibility, and privacy concerns when cloud analysis requires biomedical data uploads.
Source: pp. 2–3Method
ImJoy provides a browser-based platform with flexible plugins and browser or local and remote computation for biomedical deep-learning analysis.
Source: pp. 3, 7Results
The examples demonstrate interactive exploration of large image datasets, browser-based smartphone image classification, and user-data nuclei segmentation with real-time training monitoring.
Source: pp. 4–7Takeaways & Limitations
ImJoy’s examples show a platform for applying and adapting deep-learning tools to biomedical images and genomics across varied computational settings.
Source: p. 7The Skin-Lesion-Analyzer is intended for academic research only and has not been tested for medical use.
Source: Skin-Lesion-Analyzer (Browser) · p. 25
Abstract
from arXiv · showhide
Deep learning methods have shown extraordinary potential for analyzing very diverse biomedical data, but their dissemination beyond developers is hindered by important computational hurdles. We introduce ImJoy (https://imjoy.io/), a flexible and open-source browser-based platform designed to facilitate widespread reuse of deep learning solutions in biomedical research. We highlight ImJoy's main features and illustrate its functionalities with deep learning plugins for mobile and interactive image analysis and genomics.
Code availability
ImJoy is open-source software released under the MIT license.
Code availability
- ImJoy is open-source software released under the MIT license.The passage begins to identify where its code is hosted but does not specify the location.Source: Code availability · p. 9
Figure Legends
The figure legends show ImJoy’s browser-based architecture and examples spanning interactive image exploration, training monitoring, nuclei segmentation, and sequence analysis.
Figure 1
- ImJoy’s core is a browser app that can compute in-browser or connect to local, cluster, remote-server, and cloud plugin engines.Plugins can be shared as hyperlinks and used on different devices, including smartphones.Source: Figure Legends · p. 13
Figure 2
- HPA-UMAP displays images as a feature-based UMAP scatter plot, with click-through access to each image; the dashboard tracks loss and sample predictions during training.The dashboard’s examples show noisy input, current CARE output, and the training target.Source: Figure Legends · p. 13
- The DPNUnet plugin supports annotation, mask generation, retraining, and segmentation, while DeepBindScan predicts binding affinities for user-provided DNA or RNA sequences.DPNUnet is based on the winning entry of the 2018 Kaggle data challenge.Source: Figure Legends · p. 14
HPA-UMAP
ImJoy combines browser and native computational backends with plugins that can interact across workspaces and be chained into shareable workflows.
HPA-UMAP
- Browser plugins can use web libraries, web workers, or Python through WebAssembly, while native Python plugins use the broader scientific and deep-learning ecosystems.Conda manages virtual environments and dependent packages.Source: HPA-UMAP · p. 17
- Within a workspace, plugins can call one another’s API functions and exchange data across languages, processes, or computers.The example connects a JavaScript interface plugin to a Python data-processing plugin through the ImJoy API.Source: HPA-UMAP · p. 18
- Plugins in a workspace can be chained into workflows for more complex analysis and visualization, then stored and shared through a URL.The example workflow passes a user-entered number from one plugin to another for calculation.Source: HPA-UMAP · p. 18
Methods
The Methods section outlines ImJoy’s design and plugin system. It describes a browser-based, serverless core with optional local or remote engines, while noting that some plugins still need internet access.
Methods
- The Methods overview introduces ImJoy’s software design and plugin system and lists example plugins.It directs readers to the online documentation for a more detailed, regularly updated description.Source: Methods · p. 19
ImJoy software design
- ImJoy is a minimal, flexible browser application whose plugins provide interfaces and computation through web and native backends.The paper also describes online documentation and GitHub repositories for the app, engines, desktop app, and plugins.Source: ImJoy software design · pp. 19–20
ImJoy web application
- The static browser core sends no local data to imJoy.io, but offline use may still be limited when plugins need internet resources.The desktop app runs ImJoy locally without an internet connection; plugins may still fetch remote libraries or trained models.Source: ImJoy web application · p. 20
ImJoy plugin system
ImJoy’s plugin system combines cross-language development and execution with dependency management, isolated workspaces, and link-based distribution. Example plugins are deliberately simplified but can be adapted for other use cases.
ImJoy plugin system
- ImJoy supports browser and engine plugins, a consistent JavaScript/Python API, parallel plugin processes, and isolated workspaces for grouping task-related plugins.Plugins are individual HTML files and can be developed in ImJoy’s built-in editor.Source: ImJoy plugin system · p. 21, Example plugins · pp. 22–23
- Dependencies are resolved per plugin, with isolated virtual environments and tags for configuring operating-system, GPU, or CPU requirements.Engine plugins install their workspace, virtual environments, and software dependencies on the engine.Source: ImJoy plugin system · p. 21, Example plugins · p. 23
- GitHub repositories can serve as plugin repositories, and single-link installation can specify a commit tag to control the installed version.The paper notes that specifying a commit tag guarantees high reproducibility.Source: ImJoy plugin system · p. 21
Example plugins
- The example plugins simplify the user experience by hiding some neural-network training hyperparameters, but can be adapted, extended, or tailored to other use cases.Browser plugins can be used directly, while plugins marked “Engine” require the plugin engine.Source: Example plugins · pp. 22–23
Image Annotator plugin (Browser)
The Image Annotator is a browser plugin for manually outlining and annotating structures in multi-channel images. It supports image viewing, annotation export, and downstream processing.
Image Annotator plugin (Browser)
- The browser plugin lets users manually outline and annotate structures of interest in multi-channel TIFF or PNG images.A default image from the Human Protein Atlas opens initially; other images can be loaded through the file menu in the required folder structure.Source: Image Annotator plugin (Browser) · pp. 23–24
- Users can specify markers for structures, view channels individually or as a composite, and export GeoJSON annotations or send them to an engine for further processing.The paper gives creating masks for training segmentation algorithms as an example of further processing.Source: Image Annotator plugin (Browser) · p. 24
- Folders of images can be loaded for annotation, and images with existing annotations display those annotations automatically.The required folder organization is described in the plugin documentation.Source: Image Annotator plugin (Browser) · p. 24
HPA-UMAP (Browser)
The HPA-UMAP plugin visualizes protein-localization classifications across the Human Protein Atlas cell atlas and lets researchers explore the associated images interactively. Its embedding organizes images by feature similarity.
HPA-UMAP (Browser)
- The plugin visualizes classifications for 67,239 images from HPA cell atlas version 18 using a deep-learning method that won the 2018 Human Protein Atlas Image Classification Challenge.The UMAP projection is calculated from the final neural-network layer, a 1,024-dimensional DenseNet121 representation.Source: HPA-UMAP (Browser) · p. 24
- In the UMAP, each dot represents an image and colors indicate its human-annotated organelle localization pattern.Users can hover for a thumbnail, click a dot for a larger image and metadata, or search by gene, cell line, or localization pattern.Source: HPA-UMAP (Browser) · pp. 24–25
- The image slideshow supports inspection of the RGB composition and each of four channels individually.The green channel shows the protein of interest; blue, yellow, and red are reference-marker channels.Source: HPA-UMAP (Browser) · p. 25
Skin-Lesion-Analyzer (Browser)
The Skin-Lesion-Analyzer and HPA-Classification are browser plugins that demonstrate image classification on mobile devices and protein-localization images. The skin-lesion plugin is intended for academic research and has not been tested for medical use.
Skin-Lesion-Analyzer (Browser)
- The plugin is intended for academic research only and has not been tested for medical use.Source: Skin-Lesion-Analyzer (Browser) · p. 25
- The Skin-Lesion-Analyzer classifies skin images into seven types of potentially cancerous lesions and runs in the browser using a Keras model exported for TensorFlow.js.It can run on smartphones, where images can be taken directly with the device.Source: Skin-Lesion-Analyzer (Browser) · pp. 25–26
HPA-Classification (Browser)
- HPA-Classification predicts subcellular protein-localization patterns from images using annotated HPA cell-atlas classes and displays a Class Activation Map indicating image regions used for each class prediction.It uses a lightweight pre-trained ShuffleNet v2 network for fast browser prediction and can load images from the HPA online database.Source: HPA-Classification (Browser) · p. 26
ANNA-PALM (Engine)
The ANNA-PALM and CARE plugins provide browser-based access to deep-learning image-analysis methods, with launch pads for training or prediction and user guidance.
ANNA-PALM (Engine)
- ANNA-PALM reconstructs super-resolution views from sparse, rapidly acquired localization images and/or widefield images, with training or prediction available from a central launch pad.Users can train or predict using provided microtubule images or user-supplied ThunderSTORM-format CSV data; the launch pad links to a detailed manual recommended for new users.Source: ANNA-PALM (Engine) · p. 27
CARE (Engine)
- CARE implements deep learning-based Content-aware image restoration for denoising 3D fluorescence microscopy images.Its central launch pad supports training or prediction and links to a detailed manual recommended for new users.Source: CARE (Engine) · pp. 27–28
- In ANNA-PALM, users choose training or prediction from a central launch pad; training can use provided example data (tribolium images) or new user-provided data.Source: CARE (Engine) · pp. 27–28
DPNUnet (Engine)
DPNUnet provides a deep-learning plugin for nuclei segmentation and, together with the Image Annotator, supports a workflow from annotation through prediction.
DPNUnet (Engine)
- The plugin segments nuclei in microscopy images and is based on the winning entry of the 2018 Kaggle Data Science Bowl on nuclear segmentation.Its model uses a DPN pretrained on ImageNet with a U-net architecture.Source: DPNUnet (Engine) · p. 28
- The model combines a DPN pretrained on ImageNet with a U-net architecture.The passage identifies the DPN as pretrained on ImageNet.Source: DPNUnet (Engine) · p. 28
- Together with Image Annotator, DPNUnet supports annotating nuclei, generating masks, retraining the model, and predicting on user-supplied images.Annotations are manually outlined; generated masks serve as training targets.Source: DPNUnet (Engine) · p. 28
DeepBindScan (Engine)
DeepBindScan applies pretrained DeepBind models to user-provided DNA or RNA sequences to predict transcription-factor or RNA-binding-protein affinities. Its input-length requirement and the number of available protein models affect use and runtime.
DeepBindScan (Engine)
- DeepBindScan uses DeepBind to predict transcription-factor and RNA-binding-protein binding affinities for user-provided DNA or RNA sequences.It is implemented with Kipoi, which provides ready-to-use trained genomics models and an API.Source: DeepBindScan (Engine) · p. 29
- Each available pretrained DeepBind model predicts affinity for one transcription factor or RNA-binding protein from a specific species, using a 101-base-pair input sequence.Predictions appear in a dynamically updated vertical bar plot and can be exported as CSV files.Source: DeepBindScan (Engine) · p. 29
- For organisms with models for many proteins, such as Homo sapiens, prediction can take several minutes.Source: DeepBindScan (Engine) · p. 29