PaperlayerView exact revision on arXiv

Source-linked AI summary

A tale of perfect fit and phantom optima: how data-driven models can fail in real-time optimization

Prithvi Dake, Rahul Bindlish, James B. Rawlings

arXiv:2608.23885v1eess.SYcs.AI

TL;DR

The paper asks whether accurate data-driven process models can be trusted for economic RTO, given the cost and structural limitations of first-principles alternatives. It evaluates structured hybrid and black-box neural ODE models on a vinyl acetate benchmark and finds that accurate, stable validation fits can still yield incorrect and multiple RTO optima, with training optimization itself also causing drift. The authors therefore emphasize optimum recovery on a decision-oriented benchmark before plant application.

  • Problem

    Whether data-driven models that fit historical plant measurements accurately can be trusted for economic optimization remains unclear, especially when decision-oriented benchmarks are absent.

  • Method

    The study trains hybrid models combining known mass balances and thermodynamics with neural-network closures, alongside a fully data-driven black-box neural ODE, and tests them in RTO.

  • Results

    The trained models pass validation yet produce widely disparate RTO solutions, including many spurious optima, while stochastic training can drift from plant-optimal weights to suboptimal RTO solutions.

  • Takeaways & Limitations

    Reliable data-driven RTO models should be required to recover the plant optimum on a decision-oriented benchmark before plant testing and application.

  • Takeaways & Limitations

    The conclusions are based on simulations of a vinyl acetate benchmark, and the authors warn that process noise, unmodeled states or disturbances, and insufficiently exciting closed-loop data can worsen fragility.

Abstract

from arXiv · show

Real-time optimization (RTO) relies on process models to locate economically optimal operating conditions. Because developing first-principles models requires significant process knowledge, data-driven alternatives are increasingly attractive. Modern machine-learning models can fit historical plant data accurately and often pass standard validation tests. Whether such models can be trusted for economic optimization, however, remains unclear. We investigate this question using a vinyl acetate monomer benchmark process with a unique, well-conditioned economic optimum. We train a structured hybrid model that combines known mass balances and thermodynamics with a neural-network closure for unknown kinetics, and a fully data-driven neural ordinary differential equation (ODE) model. Both models reproduce plant measurements accurately and exhibit little variation in predictions across random initializations. Yet their economic optima differ substantially from that of the plant. Where the plant returns a single optimum on multistart search, the trained models return many phantom optima. We further show that the training optimizer alone can be yet another source of error. Even with noise-free data and initialization at weights that recover the plant optimum, stochastic gradient training can drift to weights that yield substantially worse RTO solutions. The identified model is thus an artifact of the training optimizer as well as the data. These results demonstrate that a good predictive fit of all available measurements does not guarantee reliable economic performance. A data-driven model for RTO should at least be required to recover the optimum on a decision-oriented benchmark like the one developed here before being considered for plant testing and application.

1. Introduction

RTO depends on process models whose economic reliability may not follow from accurate historical-data fits. The paper motivates decision-oriented testing because model structure, validation, and training procedures can all affect whether the plant optimum is recovered.

  • Model-based RTO: Traditional RTO reconciles model parameters with recent steady-state measurements before solving the economic optimization problem.This model-adaptation step can be constrained by steady-state wait-times and fragile detection algorithms.
  • Alternatives to model-based RTO: Model-free RTO methods avoid explicit process models but face steep evaluation costs, slow convergence, or difficulty distinguishing maxima, minima, and saddle points.Direct-search methods require increasingly many plant evaluations as the decision space grows, while feedback methods require gradient estimation and dither tuning.
  • Data-driven modeling: First-principles models extrapolate beyond operating data and remain interpretable, but they require substantial process knowledge and can be biased when their assumed structure is incomplete.Unknown reaction kinetics or fluid dynamics can prevent any parameter choice from satisfying the plant’s optimality conditions.
  • Data-driven modeling: Data-driven and physics-constrained models address limited process knowledge by learning unknown dynamics while imposing some or all known physical laws.Neural ODEs are black-box models, whereas hard-constrained approaches enforce steady-state mass and energy balances exactly.
  • Decision-oriented evaluation: Near-identical fits with tight variation across random initializations can still produce sharply different economic optima, so validation and uncertainty quantification do not certify reliable RTO performance.The paper therefore introduces a vinyl acetate benchmark with a well-conditioned optimum that models must recover rather than merely fit plant data.
  • Decision-oriented evaluation: Stochastic training can drift from weights that recover the plant optimum to substantially suboptimal RTO solutions even with correct structure and noise-free data.The paper examines this training-optimizer effect alongside structured and black-box modeling choices.

2. Industrial case-study: Vinyl acetate process

The vinyl acetate monomer benchmark represents a coupled seven-unit process with a dynamic reactor and simplified separation system, formulated for steady-state economic optimization. Its objective has a strict, well-conditioned optimum, providing a reference for comparing structured and black-box models.

  • Process configuration: The reactor is modeled as an isothermal continuous stirred-tank reactor with negligible pressure drop, while other process units operate at quasi steady state.The reactor temperature is set by the inlet-stream temperature, and component balances describe its dynamics.
  • Process configuration: The flowsheet uses simplified separation assumptions, including perfect condensation of acetic acid, water, and vinyl acetate and sharp distillation-column separation.These assumptions prevent vinyl acetate and water from being recycled to the mixer, while acetic acid is recycled.
  • Economic optimization: The RTO formulation maximizes profit from vinyl acetate revenue minus feed, recycle, purge, and processing costs while enforcing the steady-state process equations.The nominal production constraint is removed, and the liquid recycle fraction is fixed at β = 0.6 to eliminate a flat direction.
  • Process configuration: The benchmark flowsheet contains seven chemical species, seven units, and thirteen streams, with fresh feeds, recycle fractions, and reactor temperature as decision variables.The species include ethylene, ethane, acetic acid, oxygen, vinyl acetate, water, and carbon dioxide.
  • Economic optimization: The scaled reduced Hessian is negative definite with condition number 415, and multistart search detects no competing feasible optimum.The resulting optimum is locally well-conditioned with no flat directions, and profit is 7.4% of revenue.
  • Model formulations: Two model formulations are considered for RTO: a structured model enforcing mass conservation and the equation of state, and a black-box neural ODE that learns these relationships from data.The structured model uses a neural-network closure for aspects not represented by the known physical structure, whereas the black-box model approximates the ODE right-hand side directly.

3. Training and implementation details

Training data spans the RTO feasible box through pseudo-random step perturbations, while normalization, trainable output scales, and gradient controls stabilize neural ODE training.

  • Data generation: The dataset uses 200 random input trajectories, with 30% held out for validation, and perturbs feed rates and recycle fraction α by ±30%.Temperature is varied by ±10%, while β is excited in data but fixed at 0.6 during RTO.
  • Data generation: Training data spans exactly the feasible optimization box, so evaluation tests interpolation rather than extrapolation.Decision variables are held constant before each random step to provide steady-state information.
  • Optimization procedure: The implementation uses JAX, Diffrax with an adaptive ESDIRK5(4) solver, and Flax for neural-network construction and initialization.Hyperparameters include learning rate and epoch budget after fixing architecture and batch size.
  • Network scaling: The models use differentiable tanh networks because their outputs enter a nonlinear program.Inputs and outputs are scaled using dataset statistics and trainable location and scale factors.
  • Network scaling: Structured-model rate scaling separates standardized shape from physical magnitude, keeping network outputs O(1) and improving gradient conditioning.Gradient clipping is applied during ADAM, while trainable scale factors remain free to adjust rate magnitudes.

4. Results and discussion

A parametric control experiment identifies the plant rate structure and recovers its optimum under measurement noise, establishing a benchmark for data-driven models.

  • Experimental purpose: The experiment rules out uninformative data and an ill-posed optimum as trivial explanations for later model failures.The remaining question is whether the optimum remains recoverable without the correct model structure.
  • Control experiment: Tight confidence intervals show that the plant rate structure is identifiable from training data even with sensor noise.The tested relative noise levels are 0, 0.1%, and 1%.
  • Control experiment: Each parametric fit returns a single optimum under multistart RTO, matching the plant’s unique optimum structure.The single-optimum result is used as the recovery benchmark for subsequent structured and black-box models.
  • Control experiment: The profit loss is negligible at all tested noise levels, indicating that the identified parametric model recovers the plant optimum.Profit loss compares plant profit at the estimated operating point with the globally optimal plant profit.

(a) Structured models

The study trains ensembles of neural networks to assess initialization and optimizer effects, using fixed architectures and a two-stage ADAM–L-BFGS procedure.

  • Experimental design: Each of three training cases uses an ensemble of 20 networks initialized independently in weights and biases.Scaling factors are not randomly initialized, isolating neural-network parameter initialization as the varied factor.
  • Optimization procedure: Training uses 250 ADAM epochs followed by 250 L-BFGS epochs, with five fixed-order batches during ADAM.Freezing batch ordering removes shuffling as a confounding factor across initializations.
  • Evaluation: Mean-normalized RMSE is reported on training and validation measurements as error relative to mean concentration.For example, 0.02 denotes 2% error relative to mean concentrations.

(b) Black-box model

The black-box model is trained with the same overall strategy as the structured models, including controls for initialization effects and a noisy-measurement case.

  • The black-box models use the same training strategy as the structured models while removing confounding factors across 20 initializations.The cases are labeled bbox and bbox_noise, with bbox_noise using noisy measurements analogous to cmeas_noise.
  • The study trains both clean- and noisy-measurement black-box cases, with bbox_noise corresponding to 1% relative measurement noise.The noisy case is defined as analogous to cmeas_noise; the supplied passage specifies the training setup and labeling.
  • Training uses 250 ADAM epochs followed by 250 L-BFGS epochs, with mean-normalized RMSE reported for training and validation.The same passage also states NB = 5 and identifies the reported error metric.

(c) Validation

The models achieve accurate, stable validation fits, but this apparent validation success does not ensure a faithful economic landscape or reliable RTO performance. The plant has one well-conditioned optimum, whereas trained models can distort the landscape, create multiple optima, and incur substantial profit loss.

  • Validation: Excellent concentration fits and barely visible ensemble bands make structured-model predictions nearly identical across 20 random initializations.This result holds across rmeas, cmeas_rinit, cmeas, and cmeas_noise on the validation set.
  • Validation: Black-box models also achieve satisfactory validation fits with tight ensemble bands, although they are less accurate than structured models.The models therefore pass the practitioner’s stated data-based checks despite discarding physical structure.
  • Validation: Five of six validated models are useless for RTO despite low training and validation RMSE and tight prediction spread across initializations.The passage identifies the mismatch between standard validation diagnostics and economic usefulness.
  • Optimization landscape: The plant profit landscape has a single peak at (T ⋆, α⋆) and is unimodal and well-conditioned.The landscape varies reactor inlet temperature T and gas recycle fraction α while fixing the other degrees of freedom at the plant optimum.
  • Optimization landscape: rmeas reproduces the plant landscape and peak, whereas cmeas_rinit drifts from an initially recovered optimum during stochastic training.ADAM moves away from the initialization and L-BFGS recovers only part of it; a smaller learning rate merely preserves the supplied optimum.
  • Optimization landscape: cmeas, cmeas_noise, bbox, and bbox_noise distort the landscape with multiple local maxima, including a near-saddle optimum for cmeas.The near-saddle region has indefinite curvature, complicating gradient correction.
  • Economic performance under RTO: Every case except rmeas incurs significant, seed-dependent profit loss, with structured-model worst-case loss just over 30%.The same multistart search finds one plant optimum but multiple optima for visually indistinguishable models, even with noise-free structured-model data.
  • Economic performance under RTO: Black-box worst-case profit loss exceeds 50%, surpassing every structured-model case.This result directly tests whether imposing physical structure is responsible for poor economic performance.

5. Conclusions

Accurate validation fits did not ensure reliable economic optimization: trained models produced substantial profit losses and many spurious optima, while training itself could cause harmful drift. The authors therefore advocate decision-oriented optimum-recovery benchmarks and identify regularization as a possible future direction.

  • Conclusions: 30% profit loss occurred for structured models and over 50% for black-box cases in the worst case despite accurate fits and tight validation-ensemble spread.These results were obtained across model variants trained with different measurements and noise levels.
  • Conclusions: The plant yielded a single optimum under multistart search, whereas the trained models returned many spurious RTO solutions.Figure 8 compares each model’s predicted profit landscape with the plant optimum and each model’s own optimum.
  • Conclusions: Concentration-data training could move parameters away from weights that recovered the plant RTO landscape, producing a suboptimal fit.This occurred even when the optimizer was initialized at weights that recovered the plant optimum in the cmeas_rinit case.
  • Conclusions: The initial stochastic ADAM phase can leave a good parameter basin, after which L-BFGS only refines the inherited basin and does not repair the drift.The effect is invisible in training and validation fit but appears in the RTO decision.
  • Conclusions: Process noise, unmodeled states or disturbances, and insufficiently exciting closed-loop data may further worsen the identified fragility.The study reports failure even under idealized conditions with tuned hyperparameters and high-quality software.
  • Conclusions: Future work could test regularization, monotonicity constraints, and neural-network curvature bounds to improve robustness of data-driven model-based RTO.The proposed constraints target structured-model fragility and chemistry-informed rate-law behavior.

Appendix A. Process units

The appendix represents separators and splitters as linear flow splits, with species-specific and unit-specific split parameters defining the flowsheet equations.

  • Appendix A. Process units: Each separator and splitter divides inlet molar flow into a split-off stream and a remainder through a linear split.The split relation is expressed for each species entering the unit.
  • Appendix A. Process units: The species set J comprises ethylene, ethane, acetic acid, vinyl acetate, water, oxygen, and carbon dioxide.The appendix uses J to index the species participating in the flowsheet.
  • Appendix A. Process units: Each unit has its own split notation, while α and β denote recycle fractions for splitters I and II.The symbols ϕ, γ, and µ represent split fractions for the flash drum, distillation column, and absorber, respectively.

Appendix B. Parameter values

The appendix specifies the economic price vector used in the flowsheet and describes profit-loss distributions across model cases and multistart RTO seeds.

  • Appendix B. Parameter values: The price vector is (300, 33, 50, 100, 50, 50, 30, 0) $/mol for the listed stream components.The vector includes prices for p1, p2, p3, p4,Ee, p4,O, p4,A, p5, and p6.
  • Appendix B. Parameter values: Figure 10 reports relative-frequency distributions of profit loss for bbox and bbox_noise across cases, three multistart RTO seeds, and 20 ensemble members.The dotted line marks the best achievable profit loss of 0%.

Supplementary material for “A tale of perfect fit and phantom optima: how data-driven models

The supplementary material is associated with the paper on how data-driven models can fail in real-time optimization and lists its authors and affiliations.

  • Supplementary material: The supplementary material title references the paper’s focus on failure modes of data-driven models in real-time optimization.The supplied title fragment ends with the phrase “can fail in real-time optimization.”
  • Supplementary material: The listed authors are Prithvi Dake, Rahul Bindlish, and James B. Rawlings.The author line marks Dake with an asterisk.
  • Supplementary material: The affiliations are the Department of Chemical Engineering at the University of California, Santa Barbara, and Dow Chemical Company.The affiliations identify locations in California and Texas, United States.

S1 Learning-rate sweep and optimizer comparison

The supplementary sweep varies learning rate for structured and black-box training cases, comparing ADAM alone with ADAM followed by L-BFGS. Learning rates are selected by validation loss, while cmeas rinit is interpreted from its plant-optimum initialization.

  • Structured cases solve the main-paper training problem, whereas bbox and bbox noise solve the black-box training problem.
  • Every sweep fixes NB = 5 and varies only the learning rate during the stochastic ADAM phase.The Ntr training trajectories are partitioned into five batches of size Ntr/NB.
  • The four-panel figures show logarithmic training and validation losses for ADAM alone versus ADAM followed by L-BFGS.The top row reports training loss, the bottom row validation loss, the left column ADAM alone, and the right column the two-stage optimizer.
  • For cmeas, cmeas noise, bbox, and bbox noise, the selected learning rate minimizes validation loss after ADAM and L-BFGS and reaches a plateau within the epoch budget.
  • cmeas rinit uses the cmeas learning rate of 0.05 for presentation, but its key reference is the learning-rate-0 loss at initialization weights that recover the plant optimum.The dash-dotted horizontal line represents the loss before any training step.
Loading 2608.23885v1…