PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyperparameter selection for equation learning with biologically-informed neural networks

William Lavery, Jodie A. Cochrane, John T. Nardini, Sara Hamis

arXiv:2610.02954v1cs.LG

TL;DR

BINN hyperparameters interact, while the unknown coefficient functions cannot be checked against held-out ground truth, and selection remains understudied. This paper proposes an architecture-guided diagnostic workflow and tests it on synthetic systems. In noise-free cases, validation loss and learned-function consistency identify settings that recover functions within 5% of ground truth at reasonable training cost.

  • Problem

    BINN hyperparameters interact because state and RHS functions are learned simultaneously, while the unknown coefficient functions lack ground-truth validation and selection remains understudied.

    Source: §1.2 · pp. 2–3

  • Method

    The workflow selects state-network capacity, RHS-network capacities, and early-stopping patience semi-sequentially, following the BINN architecture and training-time signals.

    Source: §3.2 · p. 8

  • Results

    In noise-free synthetic cases, validation loss and learned-function consistency selected settings recovering diffusion and growth functions within 5% of ground truth at reasonable training cost.

    Source: §5 · p. 16

  • Takeaways & Limitations

    Use a substantially wider state MLP than RHS MLPs, scale RHS widths to term complexity, and select stopping patience using validation loss and learned-term consistency.

    Source: §5 · pp. 16–17

    The workflow is validated against synthetic ground truth at noise levels up to 5%, leaving its behavior under higher noise and real-measurement confounders open.

    Source: §5 · p. 17

Abstract

from arXiv · show

Biologically-informed neural networks (BINNs) have emerged as a flexible subclass of physics-informed neural networks (PINNs) for learning terms in partial differential equations from data. BINNs are particularly suited for biological systems, where the governing equations are highly nonlinear and only partially known a priori, and where data observations are often sparse, noisy, and incomplete. However, applying BINNs effectively in practice depends critically on hyperparameter selection, which remains a central challenge in equation-learning frameworks. Hyperparameters are often chosen heuristically and only cursorily documented, which limits the reproducibility of results and the transferability of methods. We present a diagnostic workflow for hyperparameter selection that can be used when the ground-truth equations are not known. The workflow is guided by three main questions: (1) Are the benefits of greater network capacity worth the cost? (2) Do more training epochs keep reducing the validation loss? (3) Do the learned terms stop changing as network capacity and training increase? We apply our workflow to synthetic systems of varying complexity with known ground truth, spanning diffusion and growth right-hand side terms and data ranging from 1D+t to 2D+t. We demonstrate that the validation loss generally follows the true error in the learned terms and distil practical rules of thumb for selecting hyperparameters in the BINN architecture. By providing a structured workflow, practical guidelines, and suggested starting values for hyperparameter selection, this work lowers the barrier to BINN-based equation learning.

1 Introduction

Equation learning infers governing equations from data, and BINNs learn unknown right-hand-side coefficient functions while retaining a known operator template. Their interacting hyperparameters are difficult to select systematically, motivating a ground-truth-free workflow and practical guidelines.

1.1 Equation learning with neural networks

  • BINNs learn unknown right-hand-side coefficient functions while retaining the known operator template, complementing equation-learning methods that infer governing equations from data.This structure is intended for mechanistic learning when data are sparse, noisy, and partially observed.Source: §1.1 · p. 2

1.2 Hyperparameter selection

  • BINN hyperparameters interact because state and right-hand-side functions are learned simultaneously, while the coefficient functions are unobserved and selection remains understudied.The resulting lack of ground-truth validation limits reproducibility and raises the barrier to use.Source: §1.2 · pp. 2–3
  • The proposed workflow selects state- and RHS-network capacities and early-stopping patience without ground truth, fixing hyperparameters in an architecture-dictated order.The workflow focuses primarily on network width and depth and the patience controlling training duration.Source: §1.2 · p. 3
  • The workflow is evaluated on synthetic systems spanning diffusion and growth terms, noisy data, and 2D+t settings, yielding transferable rules for network widths and stopping patience.The guidance favors a wider state MLP, narrow RHS MLPs scaled to term complexity, and stopping decisions based on validation loss and learned-function consistency.Source: §1.2 · p. 3

2 Background

The BINN jointly learns a state representation and right-hand-side coefficient functions while fitting data and enforcing the governing PDE and optional biological constraints. Training uses collocation points for PDE residuals and held-out grid points for validation.

2.1 The biologically-informed neural network architecture

  • A state MLP maps spatiotemporal coordinates to a density surrogate, which feeds separate diffusion and growth MLPs; all network parameters are jointly trained.The architecture represents the solution and unknown coefficient functions with learned surrogates.Source: §2.1 · p. 3

2.2 Biologically-informed learning

  • The objective combines data fitting, PDE residual, and biological-constraint losses, with the PDE residual evaluated at repeatedly sampled collocation points.The data loss uses standard least squares; the paper reports that it outperforms the GLS loss used in Lagergren et al..Source: §2.2 · p. 5, PDE loss · p. 5, Data loss · p. 5
  • Biological penalties encode assumed monotonicity and bounds for diffusion and per-capita growth across an equally spaced grid of observed density values.Diffusion is assumed non-decreasing and bounded, while per-capita growth is assumed non-increasing and bounded.Source: PDE loss · p. 6

2.3 Training–validation split

  • Individual grid points are randomly assigned to training and validation sets, supporting validation even when only one experimental realization is available.The 80/20 split monitors overfitting, and the learned RHS terms are evaluated over the observed state range rather than extrapolated to unseen regimes.Source: §2.3 · pp. 6–7

3 Methods: hyperparameter-selection workflow

The methods separate hyperparameters to tune for each data and equation setting from choices set by defaults or domain knowledge. A semi-sequential workflow uses cost, training dynamics, and learned-function consistency to guide selections, and is demonstrated on biologically plausible synthetic data.

3.1 Hyperparameters in BINNs

  • The workflow tunes MLP capacities and early-stopping patience for each data and RHS structure, while generic settings and domain-informed choices are handled separately.The first group is set once; the second is selected using domain knowledge and practical considerations, with checks described in supplementary material.Source: §3.1 · pp. 7–8

3.2 Diagnostic hyperparameter-selection workflow

  • Because capacity and patience interact, the workflow selects state-network capacity first, then RHS capacities, then early-stopping patience, confirming earlier choices as downstream settings change.Configurations are run across three random seeds to account for variability from training–validation splits.Source: §3.2 · p. 8
  • Cost-performance charts locate capacity saturation, training-dynamics plots show whether loss differences persist, and consistency plots identify when learned RHS functions stop changing.Runtime comparisons are intended only within individual case studies because machine load varies.Source: §3.2 · p. 8

3.3 Synthetic data generation

  • Synthetic ground-truth functions are restricted to biologically admissible diffusion and growth ranges informed by scratch-assay observations and Lagergren et al..The bounds are (0, 0.1) mm2 days−1 for diffusion and (−0.48, 2.4) days−1 for growth.Source: §3.3 · p. 8, Table 1 · p. 9
  • Simulations use no-flux boundaries, fine-grid solutions, then binned spatial and discretely sampled temporal data to reflect microscopy resolutions in 1D+t and 2D+t.Collocation-point counts balance runtime against stochasticity, while biological constraint weights are chosen to enforce constraints without impeding early training.Source: §3.3 · p. 9, Table 2 · p. 10

4 Results

Across four synthetic cases, the workflow selects capacities and stopping patience by examining validation loss, runtime, and consistency of recovered functions. The cases show these diagnostics guiding choices from constant diffusion in 1D+t to noisy coupled diffusion and growth in 2D+t.

4.1 Case I: demonstrating the full workflow (constant diffusion in 1D+t)

  • In the constant-diffusion case, Wu = 64 remained suitable across tested early-stopping and RHS widths; WD = 2 was selected as a low-cost choice, with ES = 2000 capturing function-recovery gains.Wu = 128 had excessive runtime, while increasing WD from 1 to 4 had little effect on loss or cost.Source: §4.1 · pp. 10–11
  • Depth three was used for both networks: density-network loss improved from depth two to three and plateaued from three to four at widths Wu = 64, 128, while diffusion-network loss was largely depth-insensitive.The choice balanced runtime and performance while retaining headroom for more complex RHS forms.Source: §4.1 · p. 12, Figure 6 · p. 12
  • In the synthetic constant-diffusion case, validation loss tracked diffusion-function MSE across width and training, and MSE stayed below the uniform 1% error reference for tested widths Wu = 32, 64, 128 and WD = 1, 2, 4.This supports validation loss as a ground-truth-free proxy for otherwise unobservable function error.Source: §4.1 · pp. 12–13, Figure 7 · p. 13

4.2 Case II: selecting WD by consistency (non-constant diffusion in 1D+t)

  • Across linear, quadratic, and exponential diffusion, consistency favored WD = 2 and ES = 3000 as the smallest setting with converged functions; the recovered functions matched ground truth within 5% MPE.Quadratic diffusion was stable across tested settings, while exponential diffusion stabilized across widths for ES ≥3000.Source: §4.2 · pp. 13–14, Figure 8 · p. 14

4.3 Case III: selecting early stopping for coupled functions (diffusion and growth in 1D+t)

  • For coupled diffusion and growth, the functions were generally consistent by ES = 2000, but quadratic growth required ES = 3000, which was selected conservatively.The comparison considered both learned functions simultaneously.Source: §4.3 · p. 15, Figure 9 · p. 15

4.4 Case IV: selecting ES on noisy data (diffusion and growth in 2D+t)

  • In noisy 2D+t data, validation loss plateaued after ES = 1000, and the learned diffusion and growth functions and ground-truth MSE supported selecting ES = 1000.The case used constant diffusion and linear growth with 5% MPE noise.Source: §4.4 · p. 16, Figure 10 · p. 16

5 Discussion

The discussion presents a ground-truth-free workflow whose training-time diagnostics selected useful BINN hyperparameters in synthetic tests. It distils architecture and patience rules, explains when settings may transfer, and identifies practical and validation limits.

Discussion

  • In noise-free synthetic cases, validation loss and learned-function consistency selected hyperparameters recovering diffusion and growth functions within 5% of ground truth at reasonable training cost.The authors argue that validation-loss trends track the otherwise unobservable joint error in recovered functions.Source: §5 · pp. 16–17
  • Use a substantially wider state MLP than RHS MLPs, scale narrow RHS widths to term complexity, and spend capacity on width rather than depth.Three hidden layers sufficed across networks; patience should use both validation loss and consistency of all coupled learned terms.Source: §5 · pp. 16–17
  • Crossley et al. [34] study individual optimisation choices, whereas this work orders joint capacity and early-stopping selection by architecture and extends it to noisy 2D+t data.The studies agree on network capacity and loss weighting where their scope overlaps.Source: §5 · p. 17
  • Reuse hyperparameters directly for closely related 1D+t or 2D+t reaction–diffusion problems; re-derive group 3 settings for intermediate problems and groups 2 and 3 for substantially different problems.Figure 11 marks settings that must be re-derived on the user's own data.Source: §5 · p. 17, Figure 11 · p. 18
  • The semi-sequential workflow tunes state width, RHS widths, then patience, and may miss configurations that full joint optimisation would uncover.It focuses on widths more than depths; practitioners also judge saturation and function stability qualitatively because fixed thresholds do not transfer across problems.Source: §5 · p. 17
  • Validation used synthetic ground truth rather than experimental data, with noise only up to 5%, leaving higher noise and real-measurement confounders untested.A concurrent study applies the workflow to real 2D+t microscopy data [11].Source: §5 · p. 17
  • The work aims to make BINN hyperparameter selection systematic, documented, and ground-truth-free to support wider and more reproducible biological modelling.The workflow's credibility is assessed in controlled cases where ground truth is known.Source: §5 · p. 18

A Synthetic data scenarios and BINN architecture overview

The appendix describes synthetic diffusion and reaction–diffusion cases spanning 1D+t and noisy 2D+t data, then identifies the diagnostics used to compare learned functions with known ground truth.

A Synthetic data scenarios and BINN architecture overview

  • Cases I–III use 1D+t noise-free cell-density data with cosine initial conditions, while Case IV uses noisy 2D+t data with a cosine-squared initial condition.The overview distinguishes diffusion-only cases from reaction–diffusion cases.Source: Figure A.1 · p. 23

B Ground-truth verification of the learned functions

B.1 Case II: non-constant diffusion (1D+t)

  • Case II evaluates learned diffusion-function MSE for linear, quadratic, and exponential diffusion across diffusion-network widths and early-stopping patience.The figure assigns one panel to each known diffusion form.Source: Figure A.2 · p. 24

B.2 Case III: coupled diffusion and growth (1D+t)

  • Case III presents validation loss, runtime, diffusion-function error, and growth-function error across RHS widths and early-stopping patiences for four growth forms paired with constant diffusion.Widths are tied (WD = WG); patience is fixed at 3000 epochs in the width comparison, while the patience comparison fixes both widths at 2.Source: Figure A.3 · p. 24, Figure A.4 · p. 25

B.3 Case IV: coupled diffusion and growth with noisy data (2D+t)

The appendix evaluates a noisy 2D+t reaction–diffusion case and describes supplementary comparisons that clarify training dynamics and document implementation choices.

B.3 Case IV: coupled diffusion and growth with noisy data (2D+t)

  • Case IV compares validation loss across joint RHS widths and early-stopping patiences, and shows diffusion and growth predictions at patience 3000.The ground-truth verification figure reports MSE against known D1(u) and G2(u) across patience values for WD = WG = 4.Source: Figure A.5 · p. 25, Figure A.6 · p. 26

Supplementary Material

  • The supplementary material presents ground-truth function errors alongside experimental-setting metrics to explain hyperparameter effects and documents noise, preprocessing, losses, MLP design, and early stopping.It also motivates group 2 hyperparameter choices and the data-loss objective.Source: Supplementary Material · p. 27

PDE and data loss weights

The section examines fixed PDE-loss weights in a noisy synthetic diffusion case. Increasing the weight does not systematically improve recovery and can delay training or impair the learned diffusion function.

PDE and data loss weights

  • The weight study uses noisy synthetic Case I with 1% MPE, fixed λdata = 1, and tests λPDE values of 1, 2, 10, and 100.Figure S.1 plots running-minimum validation loss and diffusion MSE across three training–validation splits, plus normalized losses and predictions for one split.Source: PDE and data loss weights · p. 28, Figure S.1 · p. 29
  • Early in training, larger λPDE impedes density recovery and slows diffusion-function recovery; reducing normalised PDE loss at λPDE = 10 does not materially improve end-of-training recovery.The authors identify accurate density matching as the main bottleneck limiting diffusion-function recovery.Source: PDE and data loss weights · p. 28
  • For λPDE = 100, diffusion predictions take approximately 4000 additional epochs to begin stabilising and still give the poorest final diffusion-function recovery.The low-density regime shows larger error, with density overestimation associated with diffusion underestimation there.Source: PDE and data loss weights · p. 28
  • Increasing λPDE does not systematically improve diffusion recovery and can slow optimisation or impair accurate recovery.The authors suggest that emphasizing the PDE term too early can hinder data fitting needed for later diffusion-function recovery.Source: PDE and data loss weights · p. 28
  • The authors retain fixed equal weights λdata = λPDE = 1 as a reasonable default, while noting that adaptive balancing can be more effective but adds computational cost and usually requires tuning.Data and PDE terms compete along a Pareto front whose shape is affected by system-parameter scaling of the residual.Source: PDE and data loss weights · p. 29

Weights in biological loss

Biological-loss weights strongly affect optimization: very large penalties can stall density fitting, while removing the bounding penalty allows diffusion learning to progress. Monotonicity still requires a suitable penalty, balancing constraint satisfaction against optimization speed.

Weights in biological loss

  • The bounding constraint is redundant in this setting, but removing the monotonicity constraint causes undesirable diffusion predictions for Wu = 16.The diffusion network starts near one while the true normalized coefficient is substantially smaller, so it moves below the upper bound without explicit enforcement.Source: Weights in biological loss · p. 30, Figure S.2 · p. 30
  • Large penalty weights create a validation-loss plateau and keep diffusion predictions near 0.10 mm2 days−1 because the density network fails to fit the data.With both weights set to 0, validation loss falls across several orders of magnitude and diffusion predictions move toward approximately 0.04 mm2 days−1 before converging.Source: Figure S.2 · p. 30, Weights in biological loss · p. 30
  • Smaller monotonicity weights (10 and 10^3) preserve healthy validation-loss profiles, whereas larger weights introduce plateaus and slow diffusion convergence.Final diffusion predictions remain similar across weights, especially for Wu = 64.Source: Figure S.3 · p. 31, Weights in biological loss · p. 31, Figure S.4 · p. 32
  • A weight of 10^6 enforces monotonicity throughout training and gives the best learned diffusion functions, but reintroduces a validation-loss plateau.The best checkpoint satisfies monotonicity to machine precision in all six width-by-split cases; lower weights can allow non-monotonicity during training for Wu = 16.Source: Figure S.3 · p. 31, Weights in biological loss · p. 32, Figure S.4 · p. 32

Number of collocation points

Increasing collocation points improves the reliability of the PDE training signal, but its benefits saturate as runtime rises. In the tested setting, 100 points offers a practical balance for 2D+t, while 1000 is used for 1D+t.

Number of collocation points

  • With nc = 10, a low empirical PDE residual can mask failure to satisfy the PDE elsewhere because the sparse, noisy training signal is unrepresentative.In this setting, the PDE component of validation loss is lower at nc = 10 than at nc = 1000, despite higher total validation loss.Source: Number of collocation points · p. 33, Figure S.5 · p. 34
  • The nc = 10 best model recovers diffusion and growth close to ground truth despite a larger data loss, but this incidental recovery is not a reliable selection basis.For nc ≥ 100, prediction quality appears limited mainly by optimization dynamics and model–data properties rather than collocation error; the study uses 100 points for 2D+t and 1000 for 1D+t.Source: Number of collocation points · p. 33, Figure S.5 · p. 34
  • Validation loss and function-recovery error are nearly indistinguishable for nc = 100 and nc = 1000, while runtime rises sharply only at nc = 1000.The nc = 10 runs reach the main loss drop later and have higher validation loss, driven by the data-loss component.Source: Number of collocation points · p. 33, Figure S.5 · p. 34

Activation functions

The activation-function comparison tests smooth functions because the BINN residual and biological penalties require derivatives, and the density network is the limiting factor for recovering diffusion and growth. SiLU performs best overall in the tested cases, particularly for function recovery at larger widths.

Activation functions

  • The tested activations are smooth because the PDE residual uses second spatial derivatives of the density network and the penalties use first derivatives of the diffusion and growth networks.The density-network approximation is the primary bottleneck, so the activation is selected on that network and shared across all three.Source: Activation functions · p. 35
  • SiLU diffusion MSE decreases steadily with epoch and width, whereas tanh error can rise again at narrower widths and sigmoid converges more slowly.The diffusion-MSE and prediction-consistency patterns are not fully apparent from validation-loss profiles alone.Source: Activation functions · p. 35, Figure S.6 · p. 36
  • SiLU combines competitive validation loss with consistent loss profiles, faster convergence, and better diffusion recovery for Wu ≥ 64; tanh is competitive but more width- and patience-sensitive, while sigmoid is weakest.SiLU is adopted as the default activation function across all three networks.Source: Activation functions · pp. 35–36, Figure S.6 · p. 36
  • SiLU and tanh have similar final validation losses through Wu = 32, but tanh plateaus beyond that width as its split-to-split spread increases; sigmoid performs substantially worse.At Wu = 64, SiLU reaches its best model faster than tanh, which requires nearly twice as many epochs.Source: Activation functions · p. 35, Figure S.6 · p. 36

Least-squares versus generalised least-squares data losses

In noisy synthetic Case I, least squares often recovers diffusion better than generalized least squares, whose prediction-dependent weighting can concentrate the fit on low-density points. The study also describes how its synthetic noisy observations are generated and calibrated.

Least-squares versus generalised least-squares data losses

  • Although GLS is statistically natural under the stated noise model when variance-scaling coefficients are uniform, LS often performs better in practice.GLS weights depend on predictions, assigning greater weight to points with smaller predicted density.Source: Least-squares versus generalised… · p. 37
  • GLS increasingly concentrates weight on low-density points, and the largest 1% of pointwise contributions account for a large proportion of its total loss.The loss ratio rises into the tens around 200 epochs, while the effective sample fraction falls further for GLS after approximately 100 epochs.Source: Least-squares versus generalised… · p. 38
  • LS yields better diffusion-function predictions than GLS at both tested early-stopping patience values, while GLS shows greater validation-loss sensitivity across splits.The validation-loss difference is driven by the data-loss component, and the LS–GLS comparison includes diffusion MSE and epoch-wise predictions.Source: Least-squares versus generalised… · p. 38, Figure S.7 · p. 39

Noise implementation

  • Synthetic noisy observations add a scaled noise term to the true density, truncate values at zero, and use a constant variance-scaling coefficient across observations.For a prescribed 2D+t MPE, the noise scale is solved numerically for each noise realization to meet the target within tolerance δ = 10^-4.Source: Noise implementation · p. 40

Ground-truth error metrics

The ground-truth metrics compare learned coefficient functions with known functions on a fixed density grid. Because they require ground truth, they verify workflow selections but do not guide selection in experimental settings.

Ground-truth error metrics

  • These metrics compare each learned coefficient function with its ground truth and are reported to verify selections made by the ground-truth-free workflow.They are unavailable in experimental settings and play no part in selecting hyperparameters.Source: Ground-truth error metrics · p. 40
  • The function comparison uses a fixed, equally spaced density grid restricted to the density range supported by the data.Terms with zero ground-truth values are excluded from the MSE summation.Source: Ground-truth error metrics · pp. 40–41
  • Every grid point contributes equally, without density-occupancy weighting, and the MSE has squared units of the relevant function scale.The 5th and 95th density percentiles shown in profile figures are indicative only.Source: Ground-truth error metrics · p. 41
  • MSE values are not directly comparable across diffusion and growth functions or across forms with different ground-truth magnitudes.Uniformly rescaled functions fp = (1 + p/100)f provide percentage-error context, and their p% MSE is marked in plots.Source: Ground-truth error metrics · p. 41

Data preprocessing

The preprocessing and training setup uses dimensionless variables and losses, MLP surrogates for density and coefficient functions, and validation-loss-based early stopping. The loss design uses standard least squares alongside PDE-residual and biological-constraint penalties.

Data preprocessing

  • For simulated data, density is generated as dimensionless u/K = O(1), so the pipeline uses U = 1 without further rescaling and dimensionalizes the density axis afterward.More generally, the workflow nondimensionalizes inputs and losses; outputs are dimensionless by construction and rescaled to physical units after training.Source: Data preprocessing · p. 41, Partial preprocessing · p. 42

Loss definitions

  • Standard least squares is used instead of GLS because density-dependent weights can become arbitrarily large, γ is difficult to estimate, and dropping log variance can favor inflated predictions.The GLS-inspired formulation assumes constant noise coefficients and approximates noise scaling with predicted density raised to γ.Source: Loss definitions · p. 43
  • The PDE residual loss is evaluated at collocation points sampled uniformly within the spatiotemporal bounds of the data.The biological penalty acts on a fixed density grid to enforce prescribed function bounds and monotonicity of diffusion and growth.Source: Loss definitions · p. 44, Biological loss · p. 45

Network design

  • The density field is represented by an MLP with three hidden layers of shared width Wu, using softplus output to keep density predictions non-negative.Its inputs are spatial coordinates and time, and the default hidden activation is SiLU.Source: Density MLP · pp. 45–46
  • The diffusion and growth coefficient functions are separate MLPs that each take the density-network output and return a scalar.Their hidden-layer widths are WD and WG, respectively.Source: Diffusion and growth MLPs · p. 46

Diffusion network

Growth network

  • The growth network uses an identity output activation, with no non-negativity constraint imposed on the growth term.Source: Growth network · p. 46

Early-stopping algorithm

  • Early stopping is configured by the three network widths and patience ES, saving the best-validation-loss model and its epoch.The algorithm can also record validation-loss history and, for simulated data, diffusion- and growth-MSE histories.Source: Early-stopping algorithm · pp. 46–47

Related papers

Loading 2610.02954v1…