PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Control of Scientific Instruments with Nonstationary Responses through Multimodal Agents

Guoyang Shi, Hongwen Li, Yiding Yu, Zhuang Fu, Siqi Ding, Zhengyuan Chen, JianGuo Chen, Hanyue Zhao, Tianyuan Liu

arXiv:2610.32370v1cs.AI

TL;DR

Scientific instruments with unobserved, history-dependent operating conditions can produce nonstationary responses, making static action–response mappings inadequate for successive decisions. The paper proposes a multimodal agent that combines deterministic waveform measurement, memory of trajectories and prior experience, and an independent action-feasibility check. In chronological replay, physical symbols improved quality-classification macro-F1 over waveform images alone across four evaluated models.

  • Problem

    Unobserved operating conditions can evolve with operational history, making responses nonstationary and static action–response mappings inadequate for decisions across shots.

    Source: §1 · p. 1, §2.3 · p. 6

  • Method

    A multimodal agent combines deterministic waveform-derived physical quantities, memory of task trajectories and prior experience, MLLM action proposals, and an independent feasibility check before execution.

    Source: §1 · p. 3

  • Results

    Physical symbols increased quality-classification macro-F1 by 0.21–0.32 relative to waveform images alone across four models in chronological replay.

    Source: §6 · p. 24

  • Takeaways & Limitations

    The framework provides an auditable, replaceable, and protection-compatible basis for other instruments with history dependence, partial observability, and costly experimentation.

    Source: §6 · p. 24

    The feasibility guarantee covers only encoded boundaries and historical exclusion regions; it does not guarantee breakdown-free discharges or cover sensor, configuration, or unmodeled errors.

    Source: §3.5.1 · p. 13

Abstract

from arXiv · show

Controlling scientific instruments requires parameters to be adjusted progressively in response to feedback from successive operations. When unobserved operating conditions evolve with operational history, responses become nonstationary, and a static mapping from actions to responses becomes inadequate for decisions across shots. Shot-to-shot tuning of neutral beam injection (NBI) ion sources in magnetic confinement fusion devices is representative of this class of problems: the conditions of electrode surfaces and vacuum are difficult to observe directly, discharge outcomes vary between shots, and experiments are costly and carry operational risks. We propose a physically grounded multimodal agent with three independently auditable components. Deterministic algorithms extract symbolic physical quantities and visual evidence from multichannel waveforms; a memory with two layers organizes trajectories from the current task, experience from earlier tasks, and procedural knowledge, from which a multimodal large language model (MLLM) generates candidate actions; and an independent action feasibility check enforces action constraints before execution. In a strictly chronological replay of 455 shots from four EXL-50U platforms, physical symbols alone outperformed waveform images alone by 0.21-0.32 in macro-F1 for quality classification across all four evaluated models. Retrieved context increased the mean agreement with historical action directions by 22.4-42.7 percentage points, whereas the contribution of visual evidence varied across models. Under supervised operation on five ion sources, the system achieved a beam-extraction rate of 86.80% across 409 executed shots. These results demonstrate the engineering feasibility of the framework for supervised sequential tuning on a physical scientific instrument.

1 Introduction

Scientific-instrument tuning becomes difficult when responses depend on unobserved, evolving operating conditions and physical experiments are costly. The paper proposes a physics-grounded multimodal agent that combines deterministic measurement, historical context, and independent action checks for NBI ion-source tuning.

Problem and motivation

  • NBI ion-source tuning can take weeks to months and hundreds to thousands of pulses, while identical control settings can produce different outcomes as unobserved conditions evolve.Electrode surfaces, vacuum cleanliness, and coating condition are difficult to measure directly, and physical exploration is costly and risky.Source: §1 · p. 2, Figure 1 · p. 2
  • The study frames the task as constrained sequential decision-making because latent-state drift, partial observability, and the lack of a high-fidelity simulator undermine a static action–response mapping.The introduction also notes that RL may require a simulator or extensive physical interaction, Bayesian optimization alone does not directly address latent-state drift, and static models miss trajectories and procedures.Source: §1 · p. 2

Proposed approach

  • The agent extracts verifiable physical quantities from waveforms, anchors waveform images with them, and uses task trajectories, historical cases, and procedures to propose the next shot’s action.An independent check evaluates parameter bounds, per-shot rates of change, and documented exclusion regions before parameter write-back.Source: §1 · pp. 3–4

Related-work framing

  • The paper characterizes representative paradigms by whether they use cross-shot context, integrate quantified local and macroscopic features, and independently check action constraints.These dimensions are defined for the deployment conditions studied, rather than as universal properties of every method in each paradigm.Source: §1 · p. 3, Table 1 · p. 3

Contributions and evaluation

  • The architecture assigns evidence integration and action proposals to the MLLM while deterministic modules handle quantitative readout and execution constraints remain with the PLC and hardware interlocks.The paper’s evaluation plan compares four foundation models and five evidence configurations in chronological replay of 455 shots from four platforms, alongside supervised execution on five ion sources.Source: §1 · p. 4

2 Problem Definition

The paper defines ion-source tuning as a partially observed, constrained sequence of shots: evolving physical conditions affect outcomes, and each shot’s measurements and history inform the next action. The task combines multiple operating targets with costly, risky physical interaction and therefore requires history-conditioned decisions separated from action feasibility checks.

2.1 Ion-Source Physics and Tuning Objectives

  • The source’s five coupled controls affect ion energy, electron emission, plasma production, and secondary-electron suppression, while unmeasured electrode and vacuum conditions can affect later discharges.The task targets discharge stability and accelerator high-voltage hold-off; the study evaluates voltage, current, and discharge waveforms, with beam-optics objectives left for future diagnostics.Source: Figure 2 · p. 4, §2.1 · p. 5

2.2 Sequential Decision-Making Formulation

  • The target vector specifies acceleration voltage, extraction-power, and minimum pulse-width goals, while the policy uses observed waveforms and context with action–observation history to choose the next shot.The extracted power metric is for comparisons under consistent operating conditions and does not represent net downstream ion-beam power; applicable target-set conditions can be omitted for tasks that do not assess them.Source: §2.2 · pp. 5–6

2.3 Problem Characteristics and Methodological Requirements

  • Because shots take 3–5 minutes, unsuitable actions can cause arcing or overcurrent, and latent state evolves unobservably, decisions must use cross-shot history and separate strategy proposals from feasibility checks.The task has no device-validated high-fidelity simulator, and latent-state evolution need not be monotonic with shot number.Source: §2.3 · p. 6

3 Method

The method turns multichannel waveforms into auditable physical symbols and combines them with waveform images and retrieved context to propose next-shot actions. Deterministic grounding, decision gating, and feedback updates divide the shot-level loop into independently inspectable stages.

3 Method

  • Deterministic algorithms extract auditable physical symbols from multichannel waveforms, combine them with visual evidence, and support context-informed policy generation under latent-state evolution.The method also separates candidate generation from constrained execution, with operator intervention for abnormal conditions or failed recovery.Source: §3 · pp. 6–7

3.1 Method Overview

  • The shot-level loop retrieves records, constructs evidence, retrieves context, generates a candidate, checks feasibility, executes accepted actions, and updates the short-term trajectory from feedback.The MLLM candidate uses symbolic state, image, textual context, current action, retrieved context, and task objective.Source: §3.1 · pp. 7–8

3.2 Symbolic Grounding

  • The grounding operator deterministically converts multichannel waveforms into flat-top statistics, channel-missing flags, breakdown events, oscillation flags, and fault flags.These independently auditable outputs provide the basis for system auditability.Source: §3.2 · p. 8, Figure 4 · p. 8

3.2.1 Observation Completeness and Identifiability Assessment

  • Missing acceleration-current or acceleration-voltage channels make corresponding waveform states unidentifiable, routing those observations to reacquisition or manual review rather than normal decisions.An abnormal waveform diagnostic indicates unsuitability for quantitative analysis, not confirmed sensor or hardware failure.Source: §3.2.1 · pp. 8–9

3.2.2 Beam Flat-Top Identification

  • The flat-top algorithm screens candidate pulses by absolute amplitude and duration, then retains the longest qualifying contiguous interval and extracts synchronized channel statistics.The minimum peak criterion is 0.8 times the largest candidate-pulse mean current; the retained interval must meet τ_min.Source: §3.2.2 · p. 9

3.2.3 Breakdown Detection

  • A breakdown is recorded only when both acceleration-current and acceleration-voltage criteria are met, distinguishing transient overcurrent events from other suppression-grid current.The method defines breakdown as a transient breakdown or overcurrent event and excludes steady-state suppression-grid current associated with beam divergence and related effects.Source: §3.2.3 · p. 9

3.2.4 High-Frequency Oscillation Identification

  • The oscillation detector negates the signal to find dips, filters by prominence and width, excludes breakdown interference by depth variation, and classifies oscillation when retained dip counts reach a threshold.Here, “high-frequency” refers to the repetition rate of dip events per unit time.Source: §3.2.4 · p. 9

3.3 Multimodal Representation with Symbolic Grounding

The system pairs a uniformly formatted annotated acceleration-current waveform image with deterministic symbolic state, linking both to the shot record. The MLLM uses the paired evidence to assess morphology and anomalies, check consistency, and propose a candidate action.

3.3 Multimodal Representation with Symbolic Grounding

  • A uniformly formatted annotated acceleration-current image is supplied alongside symbolic state, current control parameters, task context, and retrieved records for multimodal interpretation.Shot-number linkage preserves correspondence among the raw waveform, image, and symbolic state.Source: §3.3 · p. 10
  • The image provides overall discharge morphology and local anomaly locations, while symbolic state supplies quantitative measures and channel, breakdown, oscillation, and fault indicators.Deterministic completeness and fault checks run before the model is invoked.Source: §3.3 · p. 10
  • Figure 6 marks the detected pulse interval and flat-top statistics for shot 11888, and red triangles mark oscillation points for shot 12247.The displayed quantities include flat-top start, end, mean current, and width.Source: §3.3 · p. 10, Figure 6 · p. 11

3.4 Knowledge-Augmented Reasoning and Action Generation

Dual-layer memory combines recent within-task trajectories with persistent cross-task experience and procedural knowledge to provide temporal context for next-shot decisions. The multimodal planner integrates that context with current evidence, while model outputs do not automatically update long-term memory.

3.4.1 Short-Term–Long-Term Dual-Layer Temporal Memory

  • Short-term memory tracks recent shots to compare responses under similar settings, while long-term memory retains reusable historical experience and procedural knowledge across tasks.The short-term window uses the latest 10 experience units, and retrieval reads all records in that window in chronological order.Source: §3.4.1 · pp. 11–12
  • Long-term retrieval draws on manuals and procedures, historical cases, and expert rules; similar-state cases provide a nonparametric empirical prior without gradient updates.The retriever recalls relevant semantic chunks and experience units, with strict causal temporal isolation.Source: §3.4.1 · p. 12

3.4.2 Multimodal Reasoning and Action Generation

  • The planner receives symbolic state, waveform image, textual context, executed parameters, retrieved context, and task objective, then reports diagnosis, possible causes, and policy planning.Its structured output includes a five-dimensional candidate action, adjustment rationale, risk assessment, and evidence citations; long-term memory is not updated automatically from model outputs.Source: §3.4.2 · p. 12
  • If critical evidence is missing, the waveform is unidentifiable, the feasible set is empty, or a check fails, dispatch is refused and operator intervention is requested.The operator can pause and take over during supervised shot-to-shot execution.Source: §3.4.2 · p. 12

3.5 Action Feasibility Constraints, Control Execution, and Feedback

Candidate actions are checked against device bounds, change limits, suppression-voltage coupling, and historical exclusion regions before dispatch. Accepted actions execute through the PLC, and read-back measurements update the short-term trajectory; failed checks can trigger regeneration or operator intervention.

3.5.1 Action Feasible Set and Check

  • The feasible set combines PLC-permitted bounds, per-shot change limits, a suppression-voltage coupling interval, and frozen historical breakdown exclusion regions.The independent change limits are 5 kV, 0.5 V, 5 V, and 50 sccm for the four applicable action components; suppression voltage is constrained to 4%–10% of acceleration voltage.Source: §3.5.1 · p. 12
  • The feasibility check tests membership without modifying the model candidate; violations prompt regeneration with the violated conditions attached, up to three candidates before operator intervention.An empty feasible set prompts immediate operator intervention.Source: §3.5.1 · p. 13
  • The guarantee permits dispatch only for candidates passing the encoded check, but does not guarantee breakdown-free discharges or cover sensor errors, configuration errors, or unmodeled processes.The hardware interlock retains final shutdown authority.Source: §3.5.1 · p. 13

3.5.2 Control Execution and Feedback Update

  • PLC execution associates configuration, waveforms, retrieved evidence, model report, and check log by shot number, while communication failures or interlocks halt automatic triggering.Individual-shot outputs do not automatically modify the long-term knowledge base.Source: §3.5.2 · p. 13
  • The shot-level loop reads waveforms, grounds symbolic state, renders annotations, retrieves context, generates and checks a candidate, then dispatches accepted actions and reads back the next shot.The resulting experience unit is written to short-term in-task memory.Source: §3.5.2 · p. 14

4 Experimental Setup

The experiments evaluate waveform interpretation and next-shot action reproduction using chronological records from four EXL-50U platforms. A unified protocol compares model configurations and assesses classification, physical readouts, and agreement with historical action directions.

4.1 Data and Comparators

  • 455 shots from four independent EXL-50U platforms were assigned to six discharge-quality classes through joint expert review of raw waveforms.The platforms differed in power-supply characteristics, vacuum conditions, and electrode states.Source: §4.1 · p. 14
  • Model-reported physical fields were mapped to the six expert classes using prespecified rules, without using the evaluated shot’s human label.The mapping used flat-top count, breakdown flag, pulse width, and current amplitude; at least two flat tops indicated high-frequency oscillation in this dataset.Source: §4.1 · p. 15

4.2 Unified Cross-Model Evaluation Protocol

  • In chronological replay, each next-shot recommendation used only records available before that decision, and five configurations varied physical symbols, images, retrieved context, and human feedback.Baseline used waveform images alone; Full added the human-guidance and feedback protocol to +RAG.Source: §4.2 · pp. 15–16, Table 3 · p. 16

4.3 Metrics, Implementation, and Statistical Analysis

  • Evaluation distinguished direct quality-classification macro-F1 from field-derived six-class macro-F1 and separately assessed quantitative readouts, operator outputs, and action-direction agreement.Action reproduction used increase, hold, and decrease classes across five parameters, reporting directional accuracy separately for each and as their arithmetic mean.Source: §4.3 · pp. 16–17
  • Offline comparisons used fixed engineering configurations and model snapshots, common structured-output constraints, and no model training or fine-tuning.MDSplus handled the data interface, ChromaDB vector retrieval, and SQLite structured states and version indices.Source: §4.3 · p. 17

5 Results and Discussion

The results show nonstationary shot responses, strong offline performance from physically grounded inputs and retrieved history, and supervised operation across multiple ion sources. The authors interpret the components as complementary and modular, while identifying matched comparisons and broader evaluation as next steps.

5.1 Analysis of Nonstationarity in Device Response

  • At the primary action-neighbor threshold, 2123 pairs had a 54.7% outcome-conflict rate, and conflict was 14.5 percentage points higher for more widely separated pairs.Even pairs with identical parameters had a 44.1% conflict rate; the authors interpret the results as evidence that similar actions do not always yield similar outcomes and that response relates to operating history.Source: §5.1 · pp. 17–18

5.2 Offline Cross-Model Evaluation and Module Ablation

  • Qwen3-VL-Plus Full achieved the strongest overall offline performance, with 0.94 quality-classification macro-F1 and 95.6% mean directional agreement.Its current, pulse-width, and flat-top-count MAEs were 0.60 A, 9.60 ms, and 0.03, respectively.Source: Table 5 · p. 19, §5.2 · p. 20
  • Physical symbols improved quality-classification macro-F1 over image-only input by 0.21–0.32 across all four models.The grounding operator's pooled acceleration-current MAE was 0.68 A, and pooled main flat-top-width MAE was 10.42 ms; flat-top and breakdown event-count F1 were 97.61% and 99.66%, respectively.Source: §5.2 · pp. 18–19, Table 4 · p. 18, Table 5 · p. 19
  • Visual input had model-dependent effects: quality-classification macro-F1 rose by 0.15 for Qwen3-VL-Plus but fell by 0.06 for Qwen3.5-Plus.The paper describes physical symbols as local quantitative readings and visual evidence as global waveform structure.Source: §5.2 · pp. 19–20
  • Retrieved context increased mean directional agreement by 22.4–42.7 percentage points, with the largest and most consistent retrieval gains in action-direction reproduction.Pulse-width MAE also decreased by 8.05–22.87 ms for all four models.Source: §5.2 · pp. 19–20

5.3 Supervised On-Device Execution Validation

  • Across five ion sources, supervised operation achieved an 86.80% beam-extraction rate in 409 shots; on four sources, agent and manual records reached 86.11% and 83.83%, respectively.The four-source comparison covered 360 agent-tuned shots and 1695 historical manual-tuning shots, with an observed difference of +2.3 percentage points.Source: §5.3 · pp. 20–21, Figure 7 · p. 21
  • Across three target-oriented experiments, operating parameters increased stepwise, while the median acceleration current rose in each trajectory.The authors characterize terminal observations at higher parameter levels as transitional tuning states during continued advancement.Source: §5.3 · pp. 22–23, Figure 8 · p. 22

5.4 General Discussion and Outlook

  • The authors frame physical measurement, history-conditioned action reasoning, and feasibility checking as complementary modular roles, with interfaces that can be retained when foundation models change.They identify contemporaneous matched-task and device-state comparisons, cross-device evaluation, and additional measures as needed to characterize performance and transferability.Source: §5.4 · pp. 23–24

6 Conclusion

The conclusion presents a physics-grounded multimodal agent for tuning instruments with nonstationary responses and reports results from chronological replay and supervised on-device operation. It frames the architecture as a simulator-free, protection-compatible basis for further evaluation.

6 Conclusion

  • The agent combines deterministic waveform measurement, a two-layer memory for current and prior task context, MLLM candidate-action generation, and independent feasibility checks.The feasibility layer checks parameter bounds, per-shot rates of change, and documented exclusion regions before dispatch.Source: §6 · p. 24
  • 86.80% beam-extraction rate across 409 executed shots; chronological replay also showed physical symbols improved quality-classification macro-F1 by 0.21–0.32 over waveform images alone.On four comparable sources, supervised agent tuning achieved 86.11% versus 83.83% for manual-tuning records.Source: §6 · p. 24
  • Retrieved context increased mean directional agreement by 22.4–42.7 percentage points, while quality-outcome conflict reached 44.1% even among action-neighbor pairs with identical parameters.Conflict was 54.7% across action-neighbor pairs and increased with inter-shot separation.Source: §6 · p. 24
  • The framework is presented as a simulator-free, auditable, protection-compatible architecture and a methodological basis for other instruments with history dependence, partial observability, and costly experiments.The authors identify closed-loop benefit, sample efficiency, and cross-device generalization as subjects for subsequent quantification.Source: §6 · p. 24

Appendix A Detailed Cross-Model Results

Appendix A provides disaggregated cross-model results and a visual summary of module effects across models and configurations. The reported measures cover continuous-variable errors, quality classification, and action-direction agreement.

Appendix A Detailed Cross-Model Results

  • The appendix presents disaggregated results alongside Figure 9 as a visual summary of module effects across models and configurations.Source: Appendix A · p. 24
  • Figure 9 summarizes four measures across four models and five configurations: field-derived six-class macro-F1, mean directional agreement, quality-classification macro-F1, and acceleration-current MAE.The panels appear in that left-to-right order.Source: Figure 9 · p. 25
  • Table 8 gives detailed recognition results for the four models, reporting continuous-variable errors as MAE/RMSE and quality classification as Accuracy and macro-F1.The supplied passage identifies the reported metrics but does not provide their values.Source: Table 8 · p. 26
  • Table 9 reports directional-agreement accuracy for five parameters across the four models and gives their arithmetic mean.Source: Table 9 · p. 26

Appendix B Detailed Results for Manual and Agent Tuning

Appendix B compares manual tuning with supervised agent tuning using shot-level beam-extraction rates on four ion sources. Figure 7 and Table 10 present the comparison in visual and tabular form.

Appendix B Detailed Results for Manual and Agent Tuning

  • Figure 7 compares shot-level beam-extraction rates for manual tuning and supervised agent tuning across four ion sources.Its bars show extraction and executed-shot counts, with extraction rates above them and 95% Wilson intervals.Source: Appendix B · p. 27
  • Table 10 reports beam-extraction counts, executed-shot counts, and the difference in extraction rate for each ion source.Source: Appendix B · p. 27, Table 10 · p. 27
  • The comparison covers supervised agent tuning and manual-tuning records on four ion sources.Source: Table 10 · p. 27

Appendix C Control Parameters and Target Thresholds

Appendix C describes the agent’s action parameters and PLC-based constraints. Its adjustment limits draw on operating procedures rather than historical sample extrema alone.

Appendix C Control Parameters and Target Thresholds

  • The agent directly adjusts five parameters, with action bounds based on PLC-permitted ranges rather than narrower historical-sample extrema.Source: Appendix C · p. 27, Table 11 · p. 27
  • Operating procedures specify recommended per-shot adjustment steps for each control variable, and the upper step bound limits per-shot changes.An unchanged parameter is not subject to the interval’s lower bound.Source: Appendix C · p. 27
  • Suppression voltage is a fifth candidate action variable, constrained to 4%–10% of acceleration voltage rather than an independent fixed step size.Source: Appendix C · p. 27

Appendix D Signal-Processing Thresholds

Appendix D reports the signal-processing thresholds used by the grounding operator Φ, including platform-wide amplitude and relative-coefficient settings.

Appendix D Signal-Processing Thresholds

  • The absolute acceleration-current amplitude threshold is 1 A on all four platforms.Table 12 lists the signal-processing thresholds used by the grounding operator Φ.Source: Appendix D · p. 27, Table 12 · p. 28
  • Pulse selection uses a relative coefficient of 𝑟sel = 0.8, while flat-top localization uses 𝑟flat = 0.7.Source: Appendix D · p. 27
  • Breakdown detection combines a single current threshold, a short time window, and a voltage condition.Source: Appendix D · p. 27
Loading 2610.32370v1…