Machine Learning for Orders of Magnitude Speedup in Multi-Objective Optimization of Particle Accelerator Systems

• Simulations can be used at comparatively little computational cost.Source: tive retraining process. Our approach… · p. 2

Introduction

• The computational burden of accelerator simulations can require thousands of HPC cores and hampers comprehensive optimization.Source: Introduction · p. 2
• Multi-objective optimization explores setting trade-offs whose Pareto fronts guide working-point selection and subsequent design choices.Source: Introduction · p. 3
• High-fidelity simulations are too slow for online operation, while simplified online models trade accuracy for speed.Source: Introduction · p. 4
• Machine-learning surrogates use sparse simulation samples to predict beam parameters in fractions of a second on a laptop, with comparable accuracy.Source: Introduction · p. 4
• The models reproduce physics-simulation optimization results and support experiment planning and live optimization, while avoiding optimization runs entirely on the simulation for initial design.Source: Introduction · p. 5
• The study examines the training-sample counts needed for accurate optimization models and briefly compares model classes.Source: Introduction · p. 5
• The approach is demonstrated on the AWA injector and a more complex IsoDAR cyclotron using OPAL simulations and NSGA-II.Source: Introduction · pp. 5–6

Validation of ML Surrogate Modeling Approach for Optimization

• The comparison establishes agreement only with the GA's estimated simulation front, not the problem's true Pareto front, which neither verification nor GA convergence guarantees.Source: Results · p. 12
• With 500 random training samples, the NN produced estimated Pareto fronts close to simulation fronts; matching them required just under 66,000 physics-simulation evaluations (Fig. 4).The comparison covered the 40 nC case.Source: Results · pp. 11–12, Figure 4 · p. 13
• The NN interpolated across six input and seven output dimensions to find more optimal beam-parameter combinations outside the training data's convex hull (Fig. 5).Source: Results · pp. 13–14, Figure 5 · p. 14

Reducing Training Sample Size

• Using 500 training points caused no substantial front-accuracy loss versus 5,000, while 200 points produced substantial deviations but could still provide a warm start for iterative retraining.Source: Reducing Training Sample Size · p. 15
• Prediction improvements showed diminishing returns beyond a few thousand training samples.Source: Reducing Training Sample Size · p. 15
• Uniform random sampling of inputs did not evenly cover output space, instead producing clusters there.Source: Reducing Training Sample Size · p. 15

Iterative retraining

• Point predictions from the NN surrogate agreed excellently with the physics simulation, and GA convergence matched across surrogate and physics-simulation runs.GA convergence was also the same for models trained on physics-simulation data and on data generated by the NN surrogate.Source: Reducing Training Sample Size · pp. 16–17, Figure 9 · p. 23

Improvement in Computational Efficiency by Using ML Model

• An NN evaluation took < 1 ms on one laptop core versus 590 seconds for one 40 nC physics simulation on 8 cores, and the NN-based GA took about 2 minutes.The physics simulation had already been tuned for computational efficiency and run in parallel, so the reported savings compare against that baseline.Source: Reducing Training Sample Size · pp. 18–19
• In the 40 nC case, 132 times fewer simulation evaluations and 144 times fewer total core hours were required than with the physics simulation alone.These totals include generating training data and training the NN.Source: Reducing Training Sample Size · p. 18, Table 1 · p. 25

Training sample size

• For the 40 nC case, 500 randomly sampled training points were sufficient for an accurate estimated Pareto front; 200 points produced a less accurate front that could still seed physics-simulation fine-tuning.After a few thousand points, additional training points yielded diminishing returns.Source: Reducing Training Sample Size · pp. 15, 16, 21, Figure 6 · p. 20

Results

• For the same GA, using the NN required 3 × 10^6 times fewer computing resources; including resources to generate training data, as for initial design optimization, still gave a 144-fold core-hour improvement.Source: Reducing Training Sample Size · p. 25
• Iterative-retraining studies suggested 360 times fewer core-hours for 10 iterations, but this comparison was not repeated with the physics simulation.The authors therefore did not include iterative-retraining results in the table.Source: Reducing Training Sample Size · p. 25

Comparison with Different ML Models

• The linear SVR model's MSE was 5.5 × 10^-6 versus 3.5 × 10^-10 for the NN, a substantial difference given the small raw scales of beam quantities.For example, an emittance error of 0.1 × 10^-6 m-rad corresponds to a squared error on the order of 10^-14.Source: Comparison with Different ML Models · p. 26
• For 40 nC and 1 nC, NN MSEs were 3.5 × 10^-10 and 2.9 × 10^-8, versus PCE MSEs of 1.6 × 10^-7 and 4.5 × 10^-6, respectively.Figure 11 compares the estimated Pareto fronts using the same number of training points for the NN and PCE.Source: Comparison with Different ML Models · p. 26, Figure 11 · p. 27
• PCE provides straightforward prediction-uncertainty and sensitivity estimates via Sobol’ indices and has fewer hyperparameters, but its less-accurate fronts can seed a subsequent physics-simulation GA.A simple NN does not inherently estimate prediction uncertainty or model sensitivity without additional analysis.Source: Comparison with Different ML Models · pp. 26–27

Extension to a Challenging Cyclotron Example

• Halo predictions may be noisy and not fully converged because the computationally intensive simulation uses only 1.33 × 10^5 particles.The simulation was not tuned to establish that its mesh size and particle count were sufficient for reliable results.Source: Extension to a Challenging Cyclotron… · pp. 28–29
• The IsoDAR study predicts 12 outputs from five inputs and targets the difficult halo and particle-loss tradeoff.The five inputs are initial proton-beam current and the positions of four collimator families; the halo is especially difficult to predict.Source: Extension to a Challenging Cyclotron… · pp. 28–29
• The physics simulation was too expensive to fine-tune across the broad parameter range or run to convergence for a full multiobjective comparison.The study therefore assesses success by finding verified points near the NN-estimated front.Source: Extension to a Challenging Cyclotron… · p. 29
• Iterative retraining with verified points produces a model that fills out more of the estimated Pareto front.The process repeats optimization and verification with the retrained model.Source: Extension to a Challenging Cyclotron… · p. 30, Figure 13 · p. 32
• O(10^7) times faster execution lets the surrogate predict 12 beam-parameter outputs with reasonable accuracy.The paper illustrates prediction performance on the random-sample dataset.Source: Extension to a Challenging Cyclotron… · p. 30
• Verified points lie in close proximity to the NN-estimated Pareto front for the computationally intensive IsoDAR optimization.The physics-simulation genetic algorithm could not be run to convergence on all parameters.Source: Extension to a Challenging Cyclotron… · p. 29, Figure 13 · p. 32

Conclusions and Discussion

• Neural-network surrogates can achieve high fidelity with relatively little data, support reliable multi-objective optimization, and outperform simplified models used for fast execution.Their speed can enable more extensive optimization and larger GA populations than would be feasible with the physics simulation alone.Source: Conclusions and Discussion · pp. 35–36
• A proposed GA workflow samples the parameter space, trains a surrogate, runs the GA on it, then verifies and fine-tunes estimated Pareto points with the physics simulation.Iterative retraining can repeat this process; the surrogate can also skip costly early GA convergence stages, assuming the physical function is smooth and the model represents it well.Source: Conclusions and Discussion · pp. 35–36
• 132 times fewer simulations were needed for the 40 nC AWA case, with indications of 330–550 times fewer using iterative retraining.The surrogate estimates Pareto points quickly; the proposed process can use them to initialize a short physics-simulation GA for verification and refinement, repeating retraining and optimization until convergence.Source: Conclusions and Discussion · p. 36
• Deployment requires further work on scaling to larger systems, estimating prediction uncertainty, incorporating physics knowledge, combining simulation and measured data, and accounting for drift and model updates.The conclusions also note the challenge of reliably updating models with measured data to maintain prediction quality.Source: Conclusions and Discussion · p. 37

Incorporation into online modeling and model-based control

• Surrogates could support online setting checks, beam diagnostics, anomaly detection, model-guided optimization, and warm starts for feedback-based optimization, but these applications remain to be explored.The warm start optimizes around a forward model rather than using an inverse model, with online feedback refining suggested settings to compensate for model-machine discrepancies.Source: Conclusions and Discussion · pp. 37–38

Updating models with measured data

• Updating simulation-trained surrogates with machine measurements can account for simulation-machine deviations, but preserving simulation-derived information and effectively combining both data sources need further study.Training on simulations before machine data may expose otherwise unseen parameter-space regions and reduce reliance on machine time and archived data.Source: Conclusions and Discussion · p. 39

• Reliable updates must address machine drift and catastrophic forgetting, preserving behavior learned from previously visited configurations while incorporating new states.Source: Conclusions and Discussion · pp. 39–40

Inclusion of model uncertainty

• Reliable uncertainty estimation for neural networks remains an open research area, but could support Bayesian optimization with pre-trained models spanning broad controllable variables.Source: Conclusions and Discussion · pp. 41–42

Efficient sampling strategies

• When simulation samples are costly or variables numerous, random sampling and iterative retraining may be insufficient; the AWA random sample over- and under-represented output-space regions.Source: Conclusions and Discussion · p. 42

Scaling to higher dimension and complexity

• Scaling the method to accelerators with more inputs and outputs, wider variable ranges, or more complex beam dynamics remains an important open question.Source: Conclusions and Discussion · p. 42

Including prior physics information

• Incorporating accelerator-physics knowledge could reduce training-example requirements and improve performance in unseen parameter-space regions.Source: Conclusions and Discussion · pp. 42–43

• The authors anticipate applications in design optimization, routine prototyping, offline planning, and online machine optimization, with potential transfer between similar injector models.They suggest training on one injector and reusing the model with small updates for a similar system.Source: Conclusions and Discussion · p. 43
• The ML approach also performed on the more complicated IsoDAR cyclotron case, showing it can be applied to systems with more complex beam dynamics.Source: Conclusions and Discussion · p. 43

Appendix

• AWA data sets used 70,000 random samples at 1 nC and 80,000 at 40 nC, but only a small subset was used to train the models.The corresponding laser radii were 2 mm and 9 mm, respectively.Source: Appendix · p. 44
• For IsoDAR, the initial training set had 2,500 random points, and retraining added 100 randomly selected points from the previous estimated Pareto front.The varied inputs were initial beam current and four collimator settings.Source: Appendix · p. 44
• The IsoDAR simulation was too computationally expensive to run a multiobjective optimization over all parameters to convergence.Accordingly, the study did not run a genetic algorithm on the IsoDAR physics simulation.Source: Appendix · p. 47
• After 200 generations of 656 individuals, the neural-network Pareto front agrees with the OPAL and DEAP fronts, and its beam parameters are more optimal than those in the training set.The appendix notes that the OPAL and DEAP fronts themselves show some deviations but are in general agreement.Source: Appendix · pp. 47–48, Figure 16 · p. 49
• The reported network used four fully connected hidden layers, each with 20 nodes and hyperbolic-tangent activations, with no regularization penalties on the weights.It was trained for 10k epochs; random sample data was split into training (60%), validation (20%), and testing (20%).Source: OPAL Simulation · pp. 50–51
• The polynomial-chaos surrogate used regression with Legendre polynomials rather than the projection method.It was constructed with the Uncertainty Quantification Toolkit.Source: OPAL Simulation · p. 51
• A fourth-order PCE trained on 60% of the random sample most closely matched the NN model’s performance, following [24].The appendix describes these choices as the closest performance match to the NN model.Source: OPAL Simulation · p. 51
• Surrogate-model GA optimization used DEAP’s standard NSGA-II implementation, with hyperparameters chosen to be as close as possible to those of the OPAL GA [73].Source: OPAL Simulation · p. 51
• The research used only open-source software: OPAL, DEAP, Keras, TensorFlow, UQTk, and scikit-learn.The appendix lists references 60, 73, 67, 68, 70, and 74, respectively.Source: Code Availability · pp. 51–52