PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning Techniques for the Optimization of Target Polarization in Nuclear Physics Scattering Experiments

Armen Kasparian, Torri Jeske, Monibor Rahman, Chris Keith, James Maxwell, Thomas Britton, Malachi Schram, David Lawrence

arXiv:2610.02452v1cs.AI

TL;DR

Maintaining target polarization requires manual microwave-frequency tuning as radiation damage and material properties change, while machine learning has seen limited use in online control of nuclear physics subsystems. The paper combines probabilistic surrogate models with reinforcement learning, reporting a multisample Gaussian-process approximation with 99.4% test-set R2 and better microwave-frequency control than historical P07 operator actions.

  • Problem

    Polarization maintenance relies on expert operators' continuous monitoring and manual frequency tuning, while machine learning remains relatively limited in online control of nuclear physics experimental subsystems.

    Source: §1 · p. 1

  • Method

    The authors use surrogate models within a simulation environment to train reinforcement-learning agents for microwave-frequency control.

    Source: Reinforcement Learning · p. 5, Approach 2: Differentiable Environment · p. 14, §7 · p. 15

  • Results

    99.4% R2 on the test dataset across three samples was reported for the Gaussian-process approximation, and RL learned better frequency control than historical P07 operator actions.

    Source: Gaussian Process Approximation (GPA) · p. 8, §7 · p. 15

  • Takeaways & Limitations

    Surrogate-model environments with uncertainty supported RL agents in learning polarization-maximizing control decisions for the P07 run period.

    Source: §7 · p. 15

    Individual microwave-frequency adjustments were not logged as discrete events, making operator interventions difficult to timestamp or attribute.

    Source: Dataset & Variable Selection · p. 3

Abstract

from arXiv · show

The operation of dynamically polarized targets in nuclear physics experiments relies on continuous tuning of the microwave frequency to compensate for radiation damage and evolving material properties, a task that is traditionally performed through manual trial-and-error by expert operators. This work presents a data-driven control framework that combines surrogate modeling with reinforcement learning to optimize the target polarization. Using operational data from the APOLLO cryogenic target system, we train and evaluate multilayer perceptron and Gaussian process regression models to predict polarization as a function of microwave frequency, beam current, and accumulated radiation dose. We show that Gaussian process-based models provide calibrated uncertainty estimates and reliably identify regions outside the training distribution, while MLPs exhibit limited sensitivity to distributional shift. To enable learning and control across multiple target samples, we introduce a Gaussian process approximation and embed the surrogate model within a standardized simulation environment. A reinforcement learning agent is trained using a lower-confidence-bound reward formulation that balances performance maximization against uncertainty. We are able to show an almost 2x improvement on the operators actions utilizing our RL agent.

1 INTRODUCTION

Manual, experience-driven adjustments make sustained operation of complex experimental systems demanding, while polarization optimization in particular affects the statistical significance of spin-dependent measurements. At Jefferson Lab, radiation damage changes target behavior, and polarization is measured from NMR signals.

1 INTRODUCTION

  • Increasing target polarization improves the statistical significance of spin-dependent scattering experiments, motivating machine-learning assistance with operator optimization.The paper frames such assistance as a way to reduce the time, effort, and cost of optimization.Source: §1 · p. 1

2 POLARIZED TARGETS AT JEFFERSON LAB

  • Radiation dose degrades target polarization, so operators repeatedly tune microwave frequency to sustain polarization as sample properties evolve.Annealing can partially recover polarization, but its benefits diminish with repeated cycles.Source: §2 · p. 2

Polarization Measurements

  • For polarization measurements, the NMR background is subtracted and the polarization is extracted from the area under the NMR peak.A polynomial fit removes residual curvature before the peak area is integrated and multiplied by a calibration constant.Source: Polarization Measurements · pp. 2–3

Dataset & Variable Selection

Operator frequency adjustments are not logged as discrete events, limiting attribution of interventions and comparisons with automated systems. The models use polarization data with microwave frequency, beam current, and target dose as inputs.

Dataset & Variable Selection

  • Frequency adjustments are recorded only as measured-frequency changes, making operator interventions difficult to timestamp or attribute.The paper notes this complicates comparisons between automated systems and humans.Source: Dataset & Variable Selection · p. 3

Automated Microwave Frequency Control

  • Algorithmic approaches to automated microwave-frequency control had not yet produced a system successfully deployed in an experimental setting.The paper identifies automation as a potential way to improve data quality and reduce reliance on continuous operator attention.Source: Automated Microwave Frequency Control · p. 3

Data processing for ML

  • The single-sample dataset contains 921 P07 events, while the multisample dataset combines 5,617 events from P07, P11, and P14.Beam current, microwave frequency, and target dose are inputs; polarization is the output, and both datasets use an 80/20 train-test split.Source: Data processing for ML · p. 3
  • Input features are standardized with z-score normalization, which gives each feature zero mean and unit variance.The transformation uses each feature's mean and standard deviation.Source: Data processing for ML · p. 3

Surrogate Models

The paper uses probabilistic MLP and Gaussian-process surrogates to predict target polarization. The MLP predicts both a mean and variance, while Gaussian processes use distance-aware predictions to quantify uncertainty and identify out-of-distribution inputs.

MLPs

  • The MLP uses three input neurons, two 512-neuron hidden layers with dropout, and two outputs for the predicted mean and variance.The negative log-likelihood loss supports probabilistic regression by training the model to predict both expected value and uncertainty.Source: MLPs · p. 4

Gaussian Process (GP)

  • Gaussian processes provide probabilistic regression predictions, with predictive variance serving as a quantitative measure of model uncertainty.Their distance-aware behavior can identify inputs outside the training distribution.Source: Gaussian Process (GP) · p. 4
  • The GP kernel defines similarity between inputs and shapes the learned function's smoothness, correlation, interpolation, and uncertainty in data-sparse regions.The paper discusses radial-basis, Laplace, and Cauchy kernels.Source: Gaussian Process (GP) · p. 4

Gaussian Process Approximation (GPA)

Standard Gaussian Processes are computationally costly at scale, so Gaussian Process Approximation replaces their full covariance structure with a reduced representation. Random Fourier Features enable inference in a finite-dimensional feature space, yielding approximate posterior means and variances.

Gaussian Process Approximation (GPA)

  • Standard GP training costs O(N^3) and prediction costs O(N^2) per test point, making exact inference impractical for large datasets.N denotes the number of training samples.Source: Gaussian Process Approximation (GPA) · p. 5
  • Gaussian Process Approximation reduces computational cost by replacing the full covariance structure with a lower-dimensional approximation while retaining essential GP statistical properties.Source: Gaussian Process Approximation (GPA) · p. 5
  • Random Fourier Features approximate the kernel using D features, with W sampled from the kernel’s spectral density and b drawn element-wise from U(0, 2π).Source: Gaussian Process Approximation (GPA) · p. 5
  • The approximation permits GP inference in a finite-dimensional feature space and provides an approximate posterior mean at a test input under independent Gaussian observation noise.The corresponding approximate posterior variance is also defined for the latent function.Source: Gaussian Process Approximation (GPA) · p. 5

Reinforcement Learning

The MLP predicts target polarization closely on the test data, with R2 = 0.959, and its observed values generally lie within prediction intervals. The paper also describes an RL control workflow using TD3 and a surrogate model in a virtual environment.

Reinforcement Learning

  • The RL workflow has the agent observe a state, choose an action, receive a reward, and transition to a new state while maximizing long-term reward.Source: Reinforcement Learning · p. 5, Figure 5 · p. 6
  • The work uses TD3, whose two critic networks reduce Q-value overestimation; its surrogate system is wrapped as a Gymnasium environment for virtual training before real-system deployment.The reward evaluates action alignment with a target trajectory and may penalize large actions to encourage smooth control.Source: Reinforcement Learning · p. 5

Multilayer Perceptron (MLP)

Gaussian Process (GP)

The GP model uses an optimized Laplace kernel and provides polarization predictions with uncertainty estimates. A random-feature GPA extends prediction to multiple samples while retaining uncertainty estimates.

Gaussian Process (GP)

  • The Laplace kernel achieved lower NLL loss than the RBF and Cauchy kernels, with results largely consistent across five optimizers.The tested optimizers were Powell, Nelder–Mead, L-BFGS-B, CG, and COBYQA.Source: Gaussian Process (GP) · p. 7, Figure 10 · p. 7
  • R2 was 0.982 for the Laplace-kernel GP on test data, and its observed values fell within the roughly 95% prediction interval in most cases.The model’s average calibration had a miscalibration area of 0.037, with overconfidence in some prediction ranges and underconfidence in others.Source: Gaussian Process (GP) · pp. 7–8, Figure 11 · p. 7, Figure 12 · p. 8

Gaussian Process Approximation (GPA)

Out-of-Distribution (OOD) Behavior Analysis

Under controlled frequency shifts beyond the training range, GP and GPA uncertainty increased in the OOD region, while the MLP did not clearly distinguish ID from OOD. The models were evaluated in a standardized simulation environment based on data-driven surrogates.

Out-of-Distribution (OOD) Behavior Analysis

  • On ID test data, the MLP showed higher uncertainty than the GP.The controlled OOD test changed microwave frequency while retaining the other measured P07 input values; the training range was 140.0979–140.1822 GHz.Source: Out-of-Distribution (OOD) Behavior… · p. 9
  • As microwave frequency moved beyond the training range, GP uncertainty increased substantially, while the MLP did not clearly distinguish ID from OOD.For the GPA trained on P07, P11, and P14, uncertainty also increased substantially when frequency was shifted beyond the training distribution.Source: Out-of-Distribution (OOD) Behavior… · p. 9, Figure 16 · p. 10

Simulated Environments

  • The study used single-sample and multisample surrogate-based environment versions with identical action and observation spaces but different uncertainty estimates.The environment used Gymnasium to provide consistent state, action, and reward structures intended to preserve policy transferability.Source: Simulated Environments · pp. 9–10

Action Space

The control environment offers discrete or continuous frequency actions and an eight-dimensional observation vector. Its reward uses a lower confidence bound to trade off predicted polarization against uncertainty while discouraging large or abrupt actions.

Action Space

  • The discrete actions are to hold, increase, or decrease microwave frequency, while the continuous space enables finer frequency control; the study uses continuous actions.These options mirror the counting-house controls, with continuous actions allowing more granular control within the same range.Source: Action Space · p. 10, Table I · p. 10, Table II · p. 11

Observation Space

  • The eight-dimensional observation includes physical state, surrogate polarization mean and uncertainty, the lower confidence bound, frequency-change rate, and the previous action.Frequency is normalized to its allowable bounds, while dose and beam current are scaled; the lower confidence bound is typically computed with k = 3.Source: Observation Space · p. 11, Table III · p. 11

Reward Structure

  • The reward uses the lower confidence bound of predicted polarization and penalizes extreme actions and frequency changes to favor high polarization with low uncertainty and smoother control.The confidence multiplier is typically k = 3; the action and frequency-change penalty coefficients are 0.05 and 0.1, respectively.Source: Reward Structure · p. 11

RESULTS

DDPG, TD3, and SAC converged to comparable reward values, and TD3 was selected for the reported studies. With a single-sample P07 GP environment, the learned policy performed comparably to the human baseline without significant improvement in cumulative polarization or exploration of alternative frequency trajectories.

RESULTS

  • DDPG, TD3, and SAC converged to comparable reward values across tested models, so TD3 was selected for the subsequent studies.A differential TD3 variant was also evaluated with multisample GPA models.Source: RESULTS · pp. 11–12, Figure 17 · p. 12

Single Sample Model

  • With the single-sample P07 GP environment, the RL agent matched human-operator performance without significant improvement in cumulative polarization or alternative frequency trajectories.The comparison used a GP with a Laplace kernel trained exclusively on P07 data.Source: Single Sample Model · p. 12, Figure 18 · p. 12
  • The single-sample agent’s similar trajectories followed from the LCB reward penalizing exploration where the GP was uncertain.Because the GP was confident mainly near states observed during human operation, the agent was constrained to similar trajectories.Source: Single Sample Model · p. 12

Multi Sample Model

The multi-sample GPA broadens the modeled polarization state space but is too slow for direct RL training, motivating a lookup-table approach. With the LUT, the agent achieves higher cumulative polarization than both the human operator and the single-sample-trained agent.

Multi Sample Model

  • The multi-sample GPA generalizes the polarization state space more broadly, but its prediction time increases over 33×, making direct RL integration intractable.RL training typically requires hundreds of thousands to millions of environment interactions.Source: Multi Sample Model · p. 12

Approach 1: Precomputed Lookup Tables

  • The LUT discretizes the finite state-action space, enabling batched predictions that are numerically identical to the GPA at the same minimum frequency granularity.Generating predictions for 7,125,000 states took approximately 30 hours using batched covariance-matrix inversion.Source: Approach 1: Precomputed Lookup Tables · p. 13
  • After LUT integration, prediction time per step dropped to effectively zero.The table stores mean and standard-deviation predictions across the state space.Source: Approach 1: Precomputed Lookup Tables · p. 13
  • 33% higher cumulative polarization than both the human operator and the single-sample-trained agent was achieved by the multi-sample LUT agent.The comparison is shown in Figure 19.Source: Approach 1: Precomputed Lookup Tables · p. 13, Figure 19 · p. 13

Approach 2: Differentiable Environment

The differentiable-environment approaches use gradients through multi-step rollouts to improve policy learning, either replacing the critic or augmenting TD3. In this environment, trajectory optimization reduces episode steps but increases wall-clock time, while Differentiable TD3 does not meaningfully reduce episode steps over standard TD3.

Approach 2: Differentiable Environment

  • Trajectory Optimization replaces the critic with discounted rewards accumulated over an H-step differentiable rollout through the environment.The GP(A) model carries gradients from rewards through multi-step dynamics to the actor; H controls lookahead depth.Source: Approach 2: Differentiable Environment · p. 14
  • Training rollouts occur inside the optimization loop rather than the episode loop, so they are excluded from the episode step budget despite querying the environment H times per update.This distinction is noted as a point of contention.Source: Approach 2: Differentiable Environment · p. 14
  • Differentiable TD3 augments standard TD3 with multi-step differentiable rollouts, combining exact local environment gradients with critic estimates beyond the rollout horizon.The actor backpropagates through the rollout and bootstraps from the critic at step H.Source: Approach 2: Differentiable Environment · pp. 14–15
  • Differentiable TD3 converges in approximately the same number of episode steps as standard TD3, with greater wall-clock time from unrolling H environment steps per update.The critic alone provides adequate long-term value estimation in this environment, and the additional rollout gradients do not meaningfully accelerate step-count convergence.Source: Approach 2: Differentiable Environment · p. 15
  • Differentiable Trajectory Optimization converges in fewer episode steps than standard TD3, while Diff-TD3 has similar step counts; both incur greater wall-clock time.The relative advantage is likely environment-dependent.Source: Approach 2: Differentiable Environment · p. 15

7 CONCLUSION

The paper reports that surrogate models with uncertainty estimates can support RL agents that control microwave frequency more effectively than historical operator actions during the P07 run period. Future work will address operational integration and preparation and explore RL algorithms combining discounted long-term rewards with environment gradients.

7 CONCLUSION

  • Data-driven surrogate models with uncertainty estimates supported RL agents that learned better microwave-frequency control than historical operator actions during P07.The conclusion frames this as effective control for polarization maximization challenges in the control room.Source: §7 · p. 15
  • Future research will address operational integration and preparation and investigate RL algorithms combining discounted long-term rewards with environment gradients.Source: §7 · p. 15

Related papers

Loading 2610.02452v1…