Strategies for Deploying AI Agents in Production at Scientific User Facilities

• AIDA means AI Data Assistant; AIOps means artificial intelligence for information technology operations; ALCF means Argonne Leadership Computing Facility; APEXA means Advanced Photon EXperiment Assistant; APS means Advanced Photon Source.Source: Acronyms · p. 3
• B-PILOT means Bluesky-Plan Interface for Launch, Operation and Tracking; BELY means Best Electronic Logbook Yet; BITS means Bluesky Instrument Template Structure; BPM means beam position monitor.Source: Acronyms · p. 3
• FOFB means fast orbit feedback; HEDM means high-energy diffraction microscopy; IOC means input/output controller; IRI means Integrated Research Infrastructure; PEAR means Ptychographic Experiment and Analysis Robot; PV means process variable.Source: Acronyms · p. 3
• RAG means retrieval-augmented generation; XAS means X-ray Absorption Spectroscopy; ZMQ means ZeroMQ.Source: Acronyms · p. 3

1 Introduction

• Three converging trends—improved LLM abilities, programmable instrument-control interfaces, and MCP —make interactive control, bounded autonomy, and multimodal interpretation deployable capabilities.Source: §1 · p. 4
• Agents can handle bounded technical tasks and connect measurements to documentation and validated analysis, shortening the path to reviewable evidence while leaving scientific judgment to scientists.Source: §1 · p. 4
• Prior systems demonstrate facility assistance, voice control, teachable multimodal operation, safety-critical deployment, validated ptychography workflows, and workflows ranging from interactive to deterministic [2] [3] [5] [6] [7].Source: §1 · p. 4
• The strategy draws on APS deployments in instrument control, facility-document retrieval, and XAS, HEDM, and ptychographic analysis, with recommendations intended to be harness-independent.Source: §1 · p. 4

1.1 Target architecture

• The APS MCP/Skills Hub organizes knowledge retrieval, instrument and accelerator control, scientific analysis, and leadership computing for specialized applications and general agent harnesses.Source: §1.1 · p. 4, Figure 1 · p. 5
• The capability architecture is not a complete operational stack: safety, identity, authorization, observability, provenance, evaluation, and lifecycle learning apply across its services.Source: §1.1 · p. 5
• Network, Linux, storage, and service infrastructure underpin the capability categories; unified discovery and task contracts must preserve each underlying system’s authority.Source: §1.1 · p. 5

• Deploy the agent application where it can reach instrument interfaces, and choose an accessible LLM inference service to reduce operational complexity.Source: §2 · p. 5
• Decouple tool servers from harnesses to ease dependency management and reuse, while guarding integrations against thread-safety, asynchronous-execution, and concurrent-command hazards.Source: §2 · p. 5
• Connect systems through reviewed deterministic adapters and explicit contracts, while keeping task state, local permissions, operational outcomes, and shared evidence in their appropriate services.Source: §2 · p. 6
• Enforce physical safety in tools or control systems, retain operator override, and expose high-level instrument plans rather than sequences of low-level calls.Source: §2 · p. 6
• Provide an explicit path for tool-yielded images and other high-dimensional data when inference interfaces expect textual results.Source: §2 · p. 6
• Log tool calls and results for review, and govern proposed memories or skill updates through provenance, evaluation, expert review, versioning, approval, and rollback.Source: §2 · p. 6

3.1.1 Choosing an agent harness

• For APS beamline control needing image comprehension, start with EAA; use standard harnesses when images are unnecessary, and consider custom harnesses for unusual safety or hard real-time requirements.EAA supports vision-language reasoning, MCP tool exchange, hybrid autonomy, and optional persistent session memory, which is continuity rather than lifecycle learning.Source: §3.1.1 · p. 7
• Pi extensions can reduce facility-specific infrastructure, but require TypeScript maintenance and review because packages run with user permissions; conditional or cyclic multi-agent workflows need custom orchestration.Pi leaves MCP integration, subagents, and background shell execution outside its core; LangGraph directly represents shared state, branches, loops, and parallel execution.Source: §3.1.1 · p. 7

3.1.2 LLM inference endpoint

• Keep inference configuration provider- and, where possible, model-agnostic, with configurable endpoints, model names, and credentials; retry temporary failures and make any fallback transparent or human-approved.The suggested fallback is from a preferred endpoint to an OpenAI-compatible local model when the primary endpoint is unreachable.Source: §3.1.2 · p. 8

3.1.3 Handling tool-yielded data

• For tool-yielded images, use a shared-storage image path or intercept encoded data and pass it in a provider-recognized format; for numerical analysis, give the agent file paths and dataset metadata.Do not forward base64 image data as text; for structured files, include the dataset path, while small in-memory arrays can be saved as NPY files to reduce tool calls and context use.Source: §3.1.3 · pp. 8–9

3.1.4 Tool naming

• Because tool schemas are often flat, use self-contained tool names that identify each tool’s function and avoid name clashes.The passage’s example concerns an image-acquisition tool on an APS 2-ID-D MCP server.Source: §3.1.4 · p. 10

3.1.5 Making long tool executions non-blocking

• For long-running tools, an interim response satisfies call-response pairing and lets the agent continue; deliver completion out of band, while durable operations require persisted records and reconciliation.The in-process release mechanism does not provide recovery after process failure; reconcile with a durable execution service before resubmitting work after interruption or an ambiguous response.Source: §3.1.5 · pp. 10–11

3.1.6 Deploying computation-intensive tools

• Deploy compute-intensive tools on suitable connected hardware, isolate beamline compute with an edge device when appropriate, and consider hosted HPC inference for tasks such as image segmentation.The described 26-ID nanoprobe pattern uses a light beamline client, off-site SAM3 inference, and shared file-exchange storage for handoff.Source: §3.1.6 · pp. 11–12

3.1.7 Structured session state for task-specific workflows

• For specialized workflows, structured session records can carry experimental intent, validated controls, selected events, and processed-scan identifiers across jobs and support restart recovery.The record is persisted with validated updates and atomic replacement, but omitted information remains unavailable and endpoint changes do not guarantee equivalent model behavior; this is not lifecycle learning.Source: §3.1.7 · p. 12
• Choose between replaying message history and reconstructing context from structured state based on information needs, input length, cache reuse, and measured latency or cost.Repeatedly rewriting early prompt blocks can reduce prefix-cache reuse on endpoints that support prompt caching, even when rendered records are bounded.Source: §3.1.7 · p. 12

3.2.1 Security boundaries and least-privilege execution

• Instrument actions should pass through model-independent controls, fail closed outside permitted limits, and use common locks to prevent concurrent clients from actuating the instrument.Human confirmation, structured logging, and reviewed, versioned releases complement preventive controls.Source: §3.2.1 · p. 13
• Deterministic workflows should use validated interfaces with bounded parameters, allowlisted capabilities, and least-privilege access rather than relying on prompt instructions alone.A dedicated EPICS gateway can enforce read/write boundaries independently of the agent code.Source: §3.2.1 · p. 13

3.2.2 Tool servers and integration with control systems

• Tool servers should separate asynchronous agent communication from blocking, synchronous instrument-control calls, using a worker and ZMQ when control routines reside in the tool.Independent environments and dependencies reduce conflicts between tool servers and agent harnesses.Source: Figure 2 · p. 14, §3.2.2 · p. 14

3.2.3 Integration with existing servers and GUIs

• Existing instrument servers can replace a tool server’s worker, while a GUI application can share backend-owned state with an MCP server and expose pause, stop, or status checks.Queue Server endpoints can handle queries while a Bluesky plan runs.Source: §3.2.3 · p. 15
• B-PILOT shares Queue Server state with its optional AutoPILOT agent, while standardized BITS repositories let one tool server serve multiple APS beamlines.AutoPILOT drafts commands for user review, and BITS supplies a common registry structure and control surface.Source: §3.2.3 · pp. 15–16
• BAIT MCP relies on the Queue Server’s single-operation RunEngine to refuse writes during a running plan, but this protection does not cover other control paths or replace harness-level confirmation.Its interlock assumes the Queue Server is the instrument’s sole controller.Source: §3.2.3 · p. 16

3.2.4 Decomposing beamline procedures into skills

• Use high-level tools for bounded operations and skills for on-demand workflow documentation; keep instrument actions in controlled tools and govern skill updates through review and versioning.A skill should represent a coherent multi-step procedure, not a single hardware action.Source: §3.2.4 · p. 17

3.2.5 Worked example: a minimal nanoprobe skill set

• Bounded calibration, acquisition, and initial feature localization are supported while scientists retain control of the scientific objective, interpretation, and consequential decisions.The same nanoprobe tools and skills support an interactive session.Source: §3.2.5 · p. 18

3.3.1 Integration with deterministic analysis software

• Analysis agents should orchestrate validated software for numerical operations while the LLM plans, explains, and checks workflows, preserving interpretability and auditability.This division supports converting raw measurements into reviewable evidence without substituting the LLM for numerical methods or expert interpretation.Source: §3.3.1 · p. 18

3.3.2 Reviewable scientific artifacts and provenance

• Analysis interfaces should return reviewable artifacts—such as plots, parameter tables, fit reports, and provenance—rather than only final answers or completed-tool messages.Intermediate scientific state should be familiar to domain experts and compatible with existing practice.Source: §3.3.2 · p. 18

3.3.3 Progressive autonomy and expert oversight

• Progressive autonomy lets users approve tool calls, inspect intermediate results, adjust constraints or fit ranges, and rerun affected steps.These interactions reduce the risk that automation obscures scientific judgment.Source: §3.3.3 · p. 18

3.3.4 Worked example: XAS analysis with Dr. XAS

• Dr. XAS combines deterministic spectrum processing and fitting tools with conventional EXAFS plots, uncertainties, diagnostics, residuals, and assumptions for expert review.Displayed assumptions include fit ranges and fixed or constrained parameters.Source: §3.3.4 · p. 18

3.3.5 Worked example: HEDM analysis with APEXA

• APEXA routes MIDAS pipeline tasks to scoped specialists, limiting each specialist to relevant tools so the Motor agent cannot invoke destructive file-write or long-running-workflow tools.A deterministic keyword router dispatches queries among five specialists; sixty tools are exposed across three MCP servers.Source: §3.3.5 · p. 18

3.4.1 Operational support across service dependencies

• Investigate the affected service and its dependencies, and return supporting observations, unresolved alternatives, and the next diagnostic or operational action.An acquisition failure may require evidence from the acquisition process, host, network path, and destination storage.Source: §3.4.1 · p. 19

3.4.2 Evidence-directed diagnosis

• Evidence-directed workflows should distinguish usable evidence from unavailable instrumentation or collection failure; missing observations remain limitations, not satisfied requirements.Observations from different sampling intervals or retention periods should not be treated as simultaneous.Source: §3.4.2 · p. 20

3.4.3 Diagnostic assistance and verified recovery

• Record diagnosis, proposed change, authorized execution, and verified recovery separately; command completion alone does not establish recovery, and read-only diagnosis leaves intervention to administrators.Evaluation should distinguish diagnostic correctness, unsupported attribution, time to a useful finding, and operator effort.Source: §3.4.3 · p. 20

3.5.1 Typed and scoped memory

• KnowledgeOps covers capture, organization, retrieval, validation, reuse, revision, and retirement, with distinct memory types and scopes for different information needs.Retrieval should match the need, and shared summaries must not bypass source-record access restrictions.Source: §3.5.1 · p. 20

3.5.2 Provenance, temporal validity, and concurrent updates

• Preserve source and derivation relationships, temporal validity, and update versions so that corrected records and stale concurrent writes can be identified for revalidation.These distinctions separate what the system knew when it acted from information recorded afterward, without making an incomplete history complete.Source: §3.5.2 · p. 20

3.5.3 From operational episodes to reusable procedures

• Operational episodes can inform candidate procedures, but promotion requires evaluation, expert review, versioning, rollback, and applicability checks for the receiving environment.Records should retain attempted actions, conditions, evidence, and outcomes, including failed or inconclusive approaches.Source: §3.5.3 · p. 21

3.6.1 A common interface over existing authorities

• A common interface supports capability discovery, resource identification, bounded work, and outcome inspection without replacing native systems or concentrating privileges.Reviewed deterministic adapters translate requests to native interfaces while preserving authorization, concurrency, and operational limits; discovery alone does not establish identity, authority, or completion semantics.Source: §3.6.1 · p. 21

3.6.2 Executable contracts rather than prompt-only integration

• Repeated operations should use tested, versioned routines; deterministic code validates bounded arguments, while execution contracts distinguish proposals, approvals, actions, status, and evidence.Approvals are checked again before execution, and reported outcomes identify the operation, execution state, affected resources, and available post-action evidence.Source: §3.6.2 · pp. 21–22

3.6.3 Durable work and bounded orchestration

• Orchestration keeps task identity, dependencies, and acceptance conditions outside worker conversations, while bounding parallel work and reconciling ambiguous responses before resubmission.Partial failures require explicit states and supported recovery actions; acceptance relies on task-specific evidence, observations, and expert review rather than a worker’s completion message.Source: §3.6.3 · p. 22

3.6.4 Illustrative cross-domain investigation

• A shared-storage incident could coordinate host, network, and storage investigations, use prior incidents as hypotheses, and verify an authorized change with an independent service-level test.The example illustrates responsibility separation; it is not a report of measured autonomous recovery.Source: §3.6.4 · p. 22

4.1.2 An interactive session

• At 26-ID, EAA inspected overview images, fitted the interface, calculated its center, and issued one fine-scan call with the resulting coordinates.The deployment separates EAA, instrument control, stitching, and heavy vision inference across processes or computers, with shared NFS storage for data exchange.Source: §4.1.1 · p. 22, §4.1.2 · p. 23, Figure 5 · p. 23
• The EAA example links image evidence, a scan procedure, and a tool call; an ROI-search skill guides feature search, while zone-plate focusing is described as a further possible workflow.The paper states that this reviewable chain reduces operator attention for routine navigation and setup while preserving human authority over the scientific target and interpretation.Source: §4.1.2 · p. 23

4.2 Osprey

• Osprey exposes facility capabilities through MCP servers and a connector layer, with a gate checking commands, scope, authorization, and logs before hardware access.Its integration layer is designed to remain stable while models and harnesses can change independently.Source: §4.2 · p. 24
• Osprey’s archived BPM-timeout survey identified problematic units for repair, and an expert-checked procedure was packaged as a reusable skill.The survey counted failures over two years and distinguished transient glitches from genuine faults.Source: §4.2 · pp. 24–25
• Osprey’s on-demand diagnostics found a 60 Hz beam-motion pattern across a magnet family and a single supply above its peers, and confirmed improved orbit motion after FOFB commissioning.The outlying supply was scheduled for inspection; the FOFB comparison used similar beam conditions in two sectors.Source: §4.2 · p. 25
• Osprey runs an existing injector emittance routine end to end, returning fitted parameters and diagnostic plots without generating the scan logic itself.The paper describes deterministic use of validated routines as the more robust arrangement wherever one exists.Source: §4.2 · p. 25
• The accelerator’s single control system lacks per-instrument isolation and agent arbitration; write authority is gated, and explicit arbitration will be needed as autonomy and concurrent sessions grow.Most of the deployment is read-only, with permitted writes checked against channel whitelists and per-channel limits.Source: §4.2 · p. 25

5 Discussion and outlook

• Harnesses, endpoints, and skill sets have been changed without modifying worker-side instrument-control code, while production success should include reproducible scientific decisions and time to interpretable evidence.The authors also include knowledge transfer and operator time freed for scientific questions among the proposed success measures.Source: §5 · p. 25
• Evaluation of operational and knowledge pipelines should track evidence coverage, diagnostic correctness, time to supported findings, operator effort, recovery, escalation, and knowledge reuse.These are proposed dimensions, not measured results of a facility-wide autonomous deployment.Source: §5 · p. 25
• The foundational MCP/Skills Hub does not yet exist; its planned catalog would link users and agents to searchable tool and skill repositories with deployment guidance and API documentation.The planned GitHub organization would host separate repositories for each tool or skill.Source: §5 · pp. 25–26
• Catalog publication would use authenticated, audited promotion of fixed releases, while runtime integration would advance from read-only access to reviewed proposals and then selected authorized execution.Cross-domain workflows should define action scope, failure and cancellation behavior, arbitration, and acceptance evidence before being enabled.Source: §5 · p. 26
• APS-RAG already searches operational sources, and the proposed governed write-back queue would review corrections, annotated outcomes, and agent-proposed procedures before promoting facility knowledge.Queue records would include source evidence, applicability conditions, and review status.Source: §5 · p. 26
• Validated accelerator workflows are being packaged for reuse through a shared skill marketplace, while beamline and accelerator catalogs are intended to cross-reference capabilities for workflows spanning both domains.Cross-domain discovery should preserve boundaries, such as invoking machine-side diagnostics read-only without inheriting accelerator write access.Source: §5 · p. 26
• Analysis and computing are partially in place through EAA's offloading of SAM3 inference and Pty-Chi reconstruction to Polaris, and the authors propose common access and queryable observability across harnesses.Hutch is identified as an initial step toward observability.Source: §5 · p. 26
• Over instrument lifetimes, agents may handle routine technical work while scientists retain authority over goals and interpretation; only validated, versioned lessons should become reusable skills.The authors propose assessing success through reproducibility, access, time to trustworthy evidence, and scientific decision quality, not throughput alone.Source: §5 · pp. 26–27

Appendix A. Screenshots of user interfaces of recommended agentic harnesses

• Figures A1–A3 show EAA's WebUI, MCP tools available in Claude Code, and a scan-progress request made through an MCP tool call.Source: Figure A1 · p. 29, Figure A2 · p. 29, Figure A3 · p. 30
• Figure A4 shows the APS retrieval-augmented-generation interface, including assistant modes and an inline citation that opens its operational BELY source.Source: Figure A4 · p. 30
• Figures A5–A8 show interfaces for Dr. XAS spectroscopy analysis, PEAR ptychography reconstruction, B-PILOT with AutoPILOT, and APEXA HEDM analysis.Source: Figure A5 · p. 31, Figure A6 · p. 31, Figure A7 · p. 32
• Figures A9–A11 show ChatMD for molecular-dynamics simulations, Auto-CT for tomography reconstruction and slice manipulation, and AIDA for USAXS data-collection planning.Source: Figure A9 · p. 33, Figure A10 · p. 33, Figure A11 · p. 34