PaperlayerView exact revision on arXiv

Source-linked AI summary

From Pilots to Production: Lessons in Cross-Institutional Federated Training and Artificial Intelligence for Science

Olivera Kotevska, Max Carlson, Yan Gao, Francis Jeanson, Yijiang Li, William Lindskog, Mohammad Naseri, Minseok Ryu, Sahil Tyagi, Jerry Watkins, Feiyi Wang, Ravi Madduri, Kibaek Kim

arXiv:2609.39803v1cs.DC

TL;DR

A successful pilot does not establish a dependable multi-site system that can be governed, secured, operated, and recovered across institutions. The paper synthesizes lessons from frontier-scale federated deployments, which confirm feasibility at scale while surfacing production barriers.

  • Problem

    A trained model alone is insufficient for multi-site deployment across institutions that share neither hardware nor trust assumptions.

    Source: C. From pilots to production · p. 1

  • Method

    Two independent frontier-scale deployments are synthesized to examine federated LLM training across DOE supercomputers and distinct workflows.

    Source: §IV · p. 3

  • Results

    The deployments confirm cross-facility federated LLM training is feasible at scale and surface barriers that appear under production conditions.

    Source: §IV · p. 3

  • Takeaways & Limitations

    Production requires controls justified by threat models and release-risk checks, alongside harmonized, validated, versioned data contracts.

    Source: B. Make controls an evidence-based… · p. 5, §VI · p. 5

    The deployment base remains concentrated in DOE laboratories, with no deployment at an NSF center.

    Source: D. How far the lessons generalize · p. 4

Abstract

from arXiv · show

Many of the most valuable scientific datasets cannot be centralized: they are proprietary, export-controlled, classified, or bound by data-sovereignty restrictions. This inverts the usual paradigm: the model must move to the data, making federated artificial intelligence (AI) core infrastructure for open science. We synthesize lessons from U.S. Department of Energy national laboratories, industry deployments, and the open-source community into a structured account of moving federated AI from pilot demonstrations to dependable, multi-site production. The lessons come from five concurrent efforts spanning leadership-class supercomputers and cloud infrastructure in regulated settings. We organize them around an adapted \textit{five-domain readiness frame} and a stratified view of the stack beneath a trained model: data architecture, privacy and security controls, trust and verification, governance and socio-technical factors, and operations, the layers that decide whether a pilot becomes dependable. The question shifts from ``can we train it?'' to ``can we operate it, audit it, and change it safely?'' We report systems lessons in memory efficiency, reliability, and synchronization; set out what privacy, security, and decentralized trust require in production, and what is not yet validated there; and identify cross-cutting open problems (asynchronous federation, leakage auditing, verification standards, and harmonized data contracts) that we argue warrant a dedicated, international, open-science working group.

I. INTRODUCTION

Scientific data often cannot be centralized, so federated AI moves models across institutional boundaries while raw data stays local. The paper synthesizes cross-sector deployment lessons to frame the move from pilot training to dependable production.

A. The centralization barrier

  • Data across science, industry, and healthcare may be proprietary, export-controlled, classified, or restricted by privacy law and institutional policy.Leadership-class facilities have compute for frontier models, but the needed data remains distributed across institutions.Source: A. The centralization barrier · p. 1
  • Federated AI responds to the centralization barrier by sharing models so institutions can contribute local data signals without transferring raw data.The paper positions this approach amid work across healthcare, finance, and scientific computing.Source: A. The centralization barrier · p. 1

B. Federated AI across sectors and domains

  • Industry, open-source, and laboratory experience broadens the lessons beyond DOE, separating facility-specific constraints from recurring cross-institutional barriers.The synthesis connects regulated cloud deployments, framework developers, and national-laboratory efforts.Source: B. Federated AI across sectors and… · p. 1

C. From pilots to production

  • A successful pilot is only one artifact in a production deployment that must also be governed, secured, operated, and recovered across sites.Those institutions may share neither hardware nor trust assumptions.Source: C. From pilots to production · p. 1

D. Contributions

  • The paper synthesizes five concurrent efforts to frame readiness, report frontier-scale systems lessons, distill privacy and trust practices, and identify shared open problems.The efforts span leadership-class training and fine-tuning deployments, as well as privacy, security, and trust work.Source: D. Contributions · p. 2

II. RELATED WORK

Prior work spans cross-silo applications, leadership-class computing, privacy methods, and federated-learning frameworks. The paper sets these efforts against a production stack and readiness frame that assess both what must be built and whether a federation is ready to operate it.

II. RELATED WORK

  • Federated LLM training adds aggregation memory pressure, bandwidth constraints, and long-run reliability risk to the cross-silo setting formalized by McMahan et al. and surveyed by Kairouz et al..NNSA-laboratory training is reported in.Source: §II · p. 2
  • Cross-silo federations operate in healthcare, finance, manufacturing, neuroscience, and commercial settings, often on cloud infrastructure under health-privacy statutes.These settings provide a contrast to leadership-class systems governed by facility policy.Source: §II · p. 2
  • Leadership-class work includes fine-tuning a 7B model on 1,700+ GPUs across four DOE supercomputers [15], [16] and scalable scientific-foundation-model FL.The related work also surveys privacy challenges for science,.Source: §II · p. 2
  • Gradient-inversion and membership-inference attacks demonstrate that shared updates can leak training data, while differential privacy carries a documented utility cost –[24].Adaptive estimators improve the tradeoff but are not yet validated at frontier scale.Source: §II · p. 2
  • Existing frameworks address production, HPC integration, ecosystem breadth, or configurable topology, communication, and privacy mechanisms [26]–.These differences motivate a readiness frame for heterogeneous deployments.Source: §II · p. 2

III. A COMMON FRAME: THE PRODUCTION STACK AND FIVE READINESS DOMAINS

  • The production-stack view places the trained model above five non-training strata, making the hidden operational concerns part of a federation’s assessment.Those strata cover data architecture, privacy and security controls, trust and verification, governance and socio-technical factors, and operations.Source: §III · p. 2, Figure 1 · p. 3
  • The five-domain readiness frame assesses evidence before scaling, distinguishing production practices from pilots built on manual approvals and custom scripts.Its domains include purpose and value, agreements and approvals, data fitness, risk and justification, and run and recover.Source: §III · p. 3
  • A technical assessment alone misses coordination, process velocity, practitioner support, and incentive misalignment that shape whether data is shared.The paper argues for playbooks and sandboxes alongside APIs, and for using federation to reduce data movement while managing residual risk.Source: §III · p. 3

IV. SYSTEMS LESSONS FROM FRONTIER-SCALE FEDERATED TRAINING

Two independent frontier-scale deployments share a central aggregation server and local clients, moving model weights while keeping raw data on site. They confirm the feasibility of cross-facility federated LLM training at scale and expose barriers that arise under production conditions.

IV. SYSTEMS LESSONS FROM FRONTIER-SCALE FEDERATED TRAINING

  • Both deployments use a central aggregation server coordinating clients on site login nodes, with model weights crossing networks while raw data stays local.They span different DOE supercomputers, vendors, and workflow stacks.Source: §IV · p. 3
  • The two independent deployments confirm that cross-facility federated LLM training is feasible at scale.Source: §IV · p. 3
  • The deployments surface barriers that appear under production conditions, beyond the feasibility demonstrated by training runs.Source: §IV · p. 3

A. Deployment I: tri-laboratory training on NNSA systems

Two frontier-scale DOE deployments demonstrate cross-facility federated training across heterogeneous systems, while exposing operational constraints and limits on how far laboratory lessons generalize.

A. Deployment I: tri-laboratory training on NNSA systems

  • Both H100 and MI300A throughput scaled near-linearly to 32 nodes, with MI300A matching or slightly exceeding H100 at larger counts.The workflow was later deployed on larger classified machines and used for proprietary data across three NNSA labs.Source: A. Deployment I: tri-laboratory… · pp. 3–4, Figure 2 · p. 4

B. Deployment II: cross-facility training on ASCR systems

  • Global test loss fell monotonically from 1.39 to 0.37 over eight rounds, outperforming each facility-local model across four DOE supercomputers.The chemistry-subdomain data split reflected facility specialization, and raw data remained local.Source: B. Deployment II: cross-facility… · p. 4, Figure 3 · p. 4

C. Three systems lessons

  • Memory pressure at aggregation is tractable with quantization, tensor transfer, and file streaming; persistent state supports recovery, while synchronous communication motivates relaxed protocols.A 70B model requires approximately 130 GB per copy in 32-bit precision.Source: C. Three systems lessons · p. 4

D. How far the lessons generalize

  • Cloud-regulated deployments show memory pressure can disappear with elastic aggregation, while clients churn and least-available participants still gate rounds; the DOE-heavy evidence base limits generalization.The authors report no deployment at an NSF center and note that DOE facilities share allocation processes, security models, and often software stacks.Source: D. How far the lessons generalize · p. 4

V. PRIVACY, SECURITY, AND TRUST: WHAT PRODUCTION REQUIRES

Production federated AI requires layered controls selected against explicit threats and release risks, alongside decentralized ways to verify shared applications.

A. Defense in depth

  • Production defense asks who connects, what moves, and where it runs, matching identity controls, layered PETs, and trusted execution or attestation to those risks.The PET toolbox includes encryption, secure aggregation, differential privacy, and confidential computing.Source: A. Defense in depth · p. 5

B. Make controls an evidence-based decision

  • Select controls through a threat model, control rationale, and pre-release risk check; the tri-laboratory deployment sends only model weights across inter-site connections.Controls should match identified threats rather than be applied indiscriminately.Source: B. Make controls an evidence-based… · p. 5

C. Decentralized trust for shared apps

  • For shared apps, decentralized verification lets publishers and reviewers sign applications, hubs distribute signatures, and each site apply its own reviewer-trust policy.A stable, versioned application format makes discoverability, reuse, and “safe to run” ecosystem properties.Source: C. Decentralized trust for shared apps · p. 5

D. Where the evidence runs out

The evidence for privacy-preserving optimization remains bounded, while the shared agenda spans synchronization, leakage auditing, verification, data governance, and scaling; the authors propose open coordination.

D. Where the evidence runs out

  • Privacy controls such as differential privacy can materially reduce model utility, and proposed improvements remain demonstrated in controlled settings rather than deployed frontier-scale systems.The authors say privacy budgets should be reported plainly.Source: D. Where the evidence runs out · p. 5

VI. WHAT REMAINS UNSOLVED, AND A SHARED AGENDA

  • Asynchronous federation that preserves convergence and privacy accounting while tolerating stragglers and site failures is identified as a consequential open systems problem.Source: §VI · p. 5
  • Leakage auditing at frontier scale remains unvalidated for quantifying residual leakage from shared weights under realistic multi-site conditions and tracing it to a participant.Source: §VI · p. 5
  • The community lacks shared standards for application formats, signatures, attestation, and trust-policy expression, which could reduce duplicated security review.Source: §VI · p. 5
  • Federations need versioned, validated data contracts carrying common models, quality and drift signals, and compatibility checks rather than renegotiating them per project.The authors identify organizational infrastructure as a precondition.Source: §VI · p. 5
  • Open questions include how privacy-aware optimization methods interact at cross-facility scale and how to replace impractical full-weight exchange at trillion-parameter scale.The paper names adaptive aggregation, heterogeneous budgets, communication efficiency, low-rank adapters, sparse deltas, and revised privacy accounting.Source: §VI · p. 5
  • The authors propose a dedicated Trillion Parameter Consortium working group to coordinate the agenda, set shared evidence standards, and keep results open.Source: §VI · p. 5

ARTIFACT DESCRIPTION (AD) APPENDIX

The appendix catalogs deployment and evaluation artifacts supporting the paper’s frontier-scale case studies and privacy and trust discussions, alongside its conceptual readiness frame and open-science agenda.

A. Paper’s Main Contributions:

Part 2: Artifact Identification

  • Both H100 and MI300A deployments track close to linear per-node token-throughput scaling from 1 to 32 nodes for Llama 3.x continued pre-training.The artifact uses an 8B Llama 3.1 model and eight aggregation rounds.Source: Part 2: Artifact Identification · p. 8
  • Global test loss decreases from approximately 1.39 to 0.37 over eight aggregation rounds, consistently outperforming each facility’s local model.The deployment fine-tunes LLaMA-2 7B across four DOE ASCR supercomputers using a task-group partition of SMolInstruct.Source: Part 2: Artifact Identification · p. 8
  • 20% better final loss and 60% faster time to target accuracy are reported for FedQueue versus FedAvg, FedAsync, FedBuff, and FedCompass under heterogeneous batch-queue delays.The comparison is shown as global test loss against wall-clock time.Source: Figure 4 · p. 8, Part 2: Artifact Identification · p. 8

Artifact A3: FedQueue Queue-Aware FL Protocol:

Artifact A3 evaluates queue-aware federated aggregation through controlled simulations using empirical facility queue-delay traces.

Artifact A3: FedQueue Queue-Aware FL Protocol:

Artifact A4: TraMark Demonstrator:

Artifact A4 demonstrates TraMark, a black-box watermarking implementation for tracing client-level model leakage.

Artifact A4: TraMark Demonstrator:

  • The artifact substantiates a reference implementation of traceable black-box watermarking for client-level model-leakage attribution.Source: A. Relation to con- · p. 9
  • TraMark achieves near-perfect leakage-tracing verification without material main-task accuracy degradation.Source: A. Relation to con- · p. 9
  • The evaluation injects server-side watermarks during federated training, then queries models through a black-box API and measures verification rate.It uses ten clients with a non-IID Dirichlet partition.Source: A. Relation to con- · p. 9

Artifact A5: GDPFed Demonstrator:

Artifact A5 demonstrates heterogeneous client-level differential privacy with optimized client sampling.

Artifact A5: GDPFed Demonstrator:

  • The artifact provides a reference implementation of heterogeneous client-level differential privacy with optimized client sampling.Source: A. Relation to con- · p. 9
  • GDPFed and GDPFed+ yield higher utility than DP-FedAvg under heterogeneous per-client privacy budgets on Fashion MNIST and CIFAR-10 [44].Source: A. Relation to con- · p. 9
  • The evaluation compares P-FedAvg, DP-FedAvg, GDPFed, and GDPFed+ over matched aggregation rounds and tracks test accuracy and per-group privacy-budget consumption.Datasets are divided into three privacy-budget groups with configurable per-group ε.Source: A. Relation to con- · p. 9

ARTIFACT EVALUATION (AE) APPENDIX

The appendix traces each artifact to its primary source and records what reviewers can reproduce. It distinguishes this paper’s cross-effort synthesis from new measurements and states where independent verification is constrained.

ARTIFACT EVALUATION (AE) APPENDIX

Provenance and reproduction scope

  • A1 reproduction covers only the unclassified arXiv subset on H100- or MI300A-class hardware; production runs on El Capitan and classified NNSA machines cannot be reproduced outside those facilities.Its primary sources are Carlson et al., Carlson and Siefert, and Owens.Source: Provenance and reproduction scope · p. 9
  • A2’s full reproduction requires co-scheduled INCITE or ALCC allocations at four facilities, while a small-scale variant using production queues is public.Li et al. [15] is the primary source; APPFL releases and SMolInstruct are listed as code and data.Source: Provenance and reproduction scope · p. 9
  • A3 is fully reproduced by controlled queue simulation on one node; A4 and A5 are fully reproducible on a single GPU workstation.Their primary sources are Li et al. [16], Xu et al., and Xu et al. [44], respectively.Source: Provenance and reproduction scope · pp. 9–10

What this paper contributes to each result

  • This paper adds a cross-effort synthesis, not new measurements; A4 and A5 are reviewable implementations, with validation documented in their primary sources.A1–A3 figures are reproduced from reference deployments; their numerical evaluations, ablations, and analytic bounds are reported in primary sources.Source: What this paper contributes to each… · p. 10

Limits on independent verification

  • Two artifacts cannot be fully reproduced: A1 includes classified work and requires H100- or MI300A-class hardware for its unclassified subset; A2’s full run requires four simultaneous facility allocations.A2’s qualitative trend is reproducible at small scale without reservations, while A3, A4, and A5 are fully reproducible with a single GPU.Source: Limits on independent verification · p. 10
Loading 2609.39803v1…