PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring the Genesis Platform Capabilities to Accelerate Scientific Discovery in OPAL

Daniel Rosendo, Renan Souza, Kelsey Carter, John Lagergren, Frédéric Suter, Shelaine L. Curd, David Weston, Rafael Ferreira da Silva

arXiv:2609.08844v1cs.DC

TL;DR

At APPL, converting imagery into traits took roughly twelve hours of scientist analysis during a 40-day campaign, and results arrived only after it ended. The paper describes five Genesis capabilities composed in OPAL; in the campaign, the workflow replaced that manual burden with interactive queries.

  • Problem

    APPL’s automated image acquisition still left trait extraction to scientists, requiring roughly twelve hours of analysis and delivering results only after a 40-day campaign.

    Source: §II · p. 2

  • Method

    The paper composes five shared Genesis capabilities—managed Kubernetes, an MLflow model catalog, MAG inference, a data lakehouse, and Flowcept provenance—in the OPAL workflow.

    Source: §I · p. 1

  • Results

    In a 40-day nickel-treatment campaign, the OPAL workflow replaced roughly twelve hours of manual analysis with interactive queries.

    Source: §II · p. 2, §VIII · p. 8

  • Takeaways & Limitations

    Other use cases inherit common platform services but must supply domain models and trait code, data schema and vocabulary, and agent decision logic.

    Source: §VI · p. 7

    An outage of MAG suspends the conversational layer, although the offline pipeline continues; delayed lakehouse publication can turn seconds-long answers into minutes-long ones.

    Source: D. Limitations and Failure Modes · p. 7

Abstract

from arXiv · show

Autonomous, cross-facility science requires capabilities that no individual project should have to build for itself: managed execution for long-lived services, versioned distribution of models to remote compute systems, governed access to large language models, a shared substrate for experimental data, and end-to-end provenance. The U.S. Department of Energy Genesis Mission platform, delivered through the American Science Cloud, provides these as reusable services. This paper reports how the Genesis platform enables cross-facility experiments and accelerates scientific discovery. We explore the plant phenotyping workflow of the Orchestrated Platform for Autonomous Laboratories as the exemplar: it couples Oak Ridge National Laboratory's Advanced Plant Phenotyping Laboratory with the Frontier supercomputer. In a 40-day nickel-treatment campaign, the resulting workflow replaced roughly twelve hours of manual analysis with interactive queries returning in seconds to minutes.

I. INTRODUCTION

Cross-facility experiments have required projects to assemble difficult-to-reuse infrastructure, while plant phenotyping still depends on substantial manual trait analysis. The paper frames Genesis as shared platform capabilities for those recurring needs, alongside explicit boundaries for what remains facility- or use-case-specific.

I. INTRODUCTION

  • Cross-facility workflows have required each project to assemble infrastructure for services, model distribution, language-model access, derived data, and recording what happened.These separate solutions are difficult to reuse.Source: §I · p. 1
  • Genesis treats recurring infrastructure concerns as shared capabilities with stable interfaces, and reports five services that supply most non-domain-specific machinery for cross-facility agentic experiments.The five are managed Kubernetes execution, an MLflow model catalog, MAG inference, a data lakehouse, and Flowcept provenance.Source: §I · p. 1

II. BACKGROUND AND REQUIREMENTS

  • In a 40-day APPL campaign, converting imagery into biological traits took roughly twelve hours of scientist analysis, with results arriving after the campaign ended.Acquisition was automated, but analysis was not.Source: §II · p. 2
  • The platform requirements include managed execution, versioned model portability, governed inference, shared data, and provenance; bulk transfer and secure cross-facility messaging remain external services.These needs recur in workflows coupling instruments, HPC, and interpretation layers.Source: §II · p. 2

III. GENESIS PLATFORM & FACILITY-OPERATED CAPABILITIES EXPLORED IN OPAL

The OPAL workflow composes Genesis platform services with facility-operated infrastructure across the greenhouse, Frontier, and cloud. Managed execution, model distribution, governed inference, shared data, and provenance provide the platform-side interfaces.

III. GENESIS PLATFORM & FACILITY-OPERATED CAPABILITIES EXPLORED IN OPAL

  • The workflow moves imagery to APPL for preprocessing and then Frontier for inference, publishes derived products to the lakehouse, and uses a cloud agent, MAG, S3M Redis, and Flowcept across the components.The compute agent retrieves the model from MLflow when no local checkpoint is present.Source: §III · p. 2

A. Genesis Platform Capabilities

  • The operated Kubernetes cluster gives OPAL a network-connected place for long-lived services, while the MLflow catalog supplies versioned model artifacts to Frontier when no local checkpoint exists.The cache-with-fallback pattern makes the model version an explicit, recorded property of a run.Source: A. Genesis Platform Capabilities · pp. 2–3
  • MAG provides one inference endpoint that centralizes credentials and quota, allows model changes without agent-code changes, and makes model exchanges observable for provenance.Because every model invocation crosses the gateway, provenance coverage of reasoning is an architectural property.Source: A. Genesis Platform Capabilities · p. 3
  • The lakehouse links facility-side data production with the interpretation layer, letting the agent query published data without a Frontier account, filesystem mount, or scheduler allocation.OPAL publishes metadata, thumbnails, segmentation masks, and feature files there.Source: A. Genesis Platform Capabilities · p. 3
  • Flowcept records decisions, model exchanges, code execution, transformations, and HPC job metadata in a queryable provenance graph aligned with W3C PROV.The records persist independently of a model’s context window and span agentic and conventional steps.Source: A. Genesis Platform Capabilities · p. 3

B. Facility-operated Capabilities

OPAL relies on facility-operated services for secure cross-facility messaging and bulk data movement, while the Genesis platform takes over at publication to the lakehouse.

B. Facility-operated Capabilities

  • S3M provides token-authenticated Redis messaging across ORNL administrative and network boundaries, with both sides connecting outbound rather than exposing a new inbound service.Its security posture derives from the facility perimeter it crosses.Source: B. Facility-operated Capabilities · p. 3
  • Globus moves imagery from the APPL greenhouse to its analysis server and then to Frontier, using mature facility-configured tooling for bulk transfer.Reimplementing transfer would duplicate existing software and require tracking facility policy.Source: B. Facility-operated Capabilities · p. 3
  • The platform’s role begins when data are ready for publication, where the lakehouse takes over from transfer services.The paper distinguishes bulk movement from publishing derived data.Source: B. Facility-operated Capabilities · pp. 3–4

C. DOE Community Software

Genesis supplies capabilities common across use cases, while facility and DOE community software handles machine, perimeter, and scheduler-specific work. OPAL combines these with offline data preparation and online querying, planning, and report validation.

C. DOE Community Software

  • Academy deploys the Frontier compute agent across federated infrastructure, and Parsl separates interactive agent logic from scheduler-managed batch execution.Parsl runtime task graphs support confirmed plans of variable extent, while block provisioning packs short inference tasks into allocations.Source: C. DOE Community Software · p. 4
  • The composition principle is to use platform services for needs common across use cases and facility or community software for machine-, perimeter-, or scheduler-specific needs.The interfaces include published model versions, datasets, messages, and provenance records.Source: C. DOE Community Software · p. 4

IV. THE OPAL WORKFLOWS

  • The offline pipeline preprocesses new images, runs segmentation and trait extraction on Frontier, then publishes thumbnails, masks, feature files, and harmonized metadata to the lakehouse.The pipeline keeps traits ready before scientists ask about previously imaged plants.Source: §IV · p. 4
  • Tabular and plot queries generate code against harmonized feature data using an enriched schema and trait vocabulary, with validation and repair paths for errors.These queries take two to four model calls and three to ten seconds.Source: §IV · p. 4
  • Report validation checks a generated interpretation against a provenance card of recorded inputs, parameters, and outputs, but does not replace domain review.The check grounds claims in executed steps rather than conversation history.Source: §IV · p. 4
  • The plan path confirms a scientist’s requirements before checking for existing masks; only confirmed plans missing masks schedule targeted GPU inference.Persisted results can be found in the lakehouse on a later request.Source: §IV · p. 4, Figure 2 · p. 5

V. DEPLOYMENT AND EVALUATION

A 40-day production deployment connected APPL’s nickel-treatment campaign with Frontier through the Genesis platform and supporting services.

V. DEPLOYMENT AND EVALUATION

  • The production deployment supported a 40-day APPL campaign studying genotypic variation under nickel treatment.Trait extraction covered RGB, infrared, and fluorescence modalities.Source: A. Setup · p. 5
  • The Co-Scientist ran in Genesis Kubernetes, while an Academy compute agent on Frontier dispatched GPU inference through Parsl.Model weights were pulled from the MLflow catalog on cache miss; the Co-Scientist used MAG for inference and the lakehouse for data.Source: A. Setup · p. 5
  • Globus moved imagery from the greenhouse to Frontier in two hops, S3M Redis carried agent control messages, and Flowcept captured provenance.Source: A. Setup · p. 5

B. A Representative Session

The representative sessions use plant-area measurements to assess nickel response while also surfacing and following up on a watering-related stress event. They illustrate how scientists can query results and interrogate workflow provenance.

B. A Representative Session

  • Leaf area is the experiment’s primary nickel-tolerance trait, defined as maintaining area under high Ni stress.Source: B. A Representative Session · p. 5
  • Plant 29073 of ISU 210 retained 108 mm2 at 500 mg Ni kg−1, while genotype 9 plant 29038 reached approximately 110 mm2 at 200 mg Ni kg−1.Source: B. A Representative Session · p. 5
  • Genotypes 9 and ISU 210 appear promising, but RIL 407 needs higher-concentration testing and Ecm29’s tolerance cannot be judged without higher-Ni data.Source: B. A Representative Session · p. 5
  • A May 3 plot showed greater than 30% leaf-area collapse in five plants, most plausibly from watering failure, overriding the nickel-tolerance signal before genotype selection.The recommendation was to monitor the plants, check watering, and repeat under confirmed proper irrigation before using the data for genotype selection.Source: B. A Representative Session · p. 6
  • The plot confirms the May 3 abrupt leaf-area drops followed a watering failure; four of five plants had already recovered by now, while plant 28801 continued to decline and should be watched closely.Source: B. A Representative Session · p. 6
  • Recording every workflow step lets scientists interrogate how a result was produced across both platform and facility sides.Source: B. A Representative Session · p. 6

C. Performance

The evaluation reports faster, broader analysis than the manual campaign baseline, with low provenance overhead. Most queries use previously published results, while cold-path latency depends heavily on shared-machine scheduling.

C. Performance

  • Cold-path queries can take tens of minutes or more when batch queue waits are long, whereas the hourly offline pipeline usually leaves recent imagery available in the lakehouse.An individual inference takes on the order of ten seconds; queue wait, rather than inference, dominates cold-path cost.Source: C. Performance · p. 7
  • Provenance capture added under 1% to end-to-end runtime at the reported granularity and scale.The paper argues this supports provenance as a default platform property rather than a debugging mode.Source: C. Performance · p. 7
  • About 100× more traits per run and roughly 600× less analysis time were achieved against a campaign baseline of roughly twelve hours of manual work.The workflow extracts the full trait panel for every plant at every imaging round and completes within minutes of image acquisition.Source: C. Performance · p. 7

D. Limitations and Failure Modes

Failures are handled asymmetrically across the workflow: interpretation and HPC failures remain visible, while provenance-service outages block provenance queries and MAG outages suspend conversation.

D. Limitations and Failure Modes

  • Interpretation failures trigger one repair retry, then return the error and generated code to the scientist rather than being silently swallowed.HPC-crossing failures are reported with a provenance link showing how far the plan progressed.Source: D. Limitations and Failure Modes · p. 7
  • A provenance-service outage makes provenance questions unanswerable, motivating an availability target for provenance as platform infrastructure.Source: D. Limitations and Failure Modes · p. 7
  • A MAG outage suspends conversation, but the language-model-free offline pipeline continues; delayed lakehouse publication shifts availability checks from seconds to minutes.Source: D. Limitations and Failure Modes · p. 7

VI. GENERALIZING TO OTHER USE CASES

The platform supplies reusable capabilities, while each use case provides its domain model, data schema, and agent decision logic. Two transferable patterns are availability checks and cache-with-fallback model retrieval.

VI. GENERALIZING TO OTHER USE CASES

  • Only the conversational agent, compute-agent logic, APPL analysis server, and domain model and trait algorithms are use-case-specific; other components are inherited.Source: §VI · p. 7
  • Use cases must supply a domain model and quantity-derivation code, a schema and vocabulary, and agent logic defining complete plans and worthwhile computations.The lakehouse stores submitted data, and correct generated analysis code depends on knowing what its columns mean.Source: §VI · p. 7
  • Availability checks serve requests from published derived products before computing, while cold-node catalog lookup keeps steady-state model retrieval local and versions explicit per run.Source: §VI · p. 7

VII. RELATED WORK

Related work establishes facility-coupling and instrument-assistance patterns; this paper instead reports platform-operated capabilities and the boundary between shared infrastructure and use-case work.

VII. RELATED WORK

  • The Superfacility model and cross-facility workflow work, couple instruments with HPC, while INTERSECT pursues loosely coupled facility services.Secure programmatic facility-access work, [16] supports tractable cloud-to-HPC interaction.Source: §VII · p. 8
  • VISION and CALMS assist with instrument operation, whereas OPAL begins after acquisition and emphasizes HPC-scale analysis and provenance; earlier additive-manufacturing work used a per-project architecture.The present contribution is complementary: platform-operated capabilities rather than a facility-integration framework or per-project architecture.Source: §VII · p. 8

VIII. CONCLUSION AND FUTURE WORK

The OPAL demonstration combines five Genesis capabilities across APPL and Frontier, replacing extensive manual analysis with interactive queries. Future work targets platform integration and validation across other workflow shapes and facilities.

VIII. CONCLUSION AND FUTURE WORK

  • In a 40-day nickel-treatment campaign, interactive queries returning in seconds to minutes replaced roughly twelve hours of manual analysis, with under 1% provenance overhead.Source: §VIII · p. 8
  • OPAL couples ORNL’s APPL facility with Frontier, and only the APPL analysis server, conversational agent, compute-agent logic, and plant domain model and trait algorithms were plant-specific.The workflow lets biologists ask natural-language questions while the campaign is running.Source: §VIII · p. 8
  • Future work includes integrating Flowcept and the lakehouse with Genesis, validating boundaries for streaming instruments, and testing deployments across multiple DOE laboratories.The lakehouse integration is in progress; Flowcept is to become the platform provenance service.Source: §VIII · p. 8
Loading 2609.08844v1…