PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Frequency Decomposition Benefit Physics-Informed Neural Networks? A Preliminary Ablation Study

Shubham Rai

arXiv:2608.24940v1cs.LGcs.AI

TL;DR

PINNs face spectral bias when approximating high-frequency and multi-scale PDE solutions, while existing frequency-aware methods provide limited evidence about when they help. The paper introduces DBSG-PINN and evaluates its frequency decomposition and adaptive gating through ablations on five one-dimensional benchmarks. Benefits are strongest on spectrally complex problems, but mixed results on simpler problems and the limited study scope make the findings preliminary.

  • Problem

    Existing frequency-aware PINN methods rarely compare against matched-capacity ablations to determine whether their value depends on the PDE solution’s spectral character.

  • Method

    DBSG-PINN uses separate low- and high-frequency subnetworks with an adaptive gate, and removes each component individually in ablations across five one-dimensional PDE benchmarks.

  • Results

    Frequency decomposition helped most on spectrally complex benchmarks, cutting relative L2 error by up to 59.2% on Multimodal Wave, while 1D Wave favored NoGate by a wide margin.

  • Takeaways & Limitations

    The preliminary benchmark pattern suggests that decomposition and gating may be most useful when target solutions genuinely mix frequency regimes.

  • Takeaways & Limitations

    Results use one training seed across five one-dimensional benchmarks, and the post-hoc finite-difference PDE residual is not the autodiff residual minimized during training.

Abstract

from arXiv · show

Partial differential equations (PDEs) often have high-frequency and multi-scale features that neural networks struggle to approximate. Physics-Informed Neural Networks (PINNs) build the governing equations directly into training, but suffer from spectral bias: they learn low-frequency components faster than high-frequency ones. Techniques such as Fourier feature embeddings and sinusoidal activations address this, but most studies assume they help across the board without checking which spectral regimes actually benefit. We introduce a dual-branch, spectrally-gated architecture (DBSG-PINN) that splits low- and high-frequency components into separate subnetworks joined by an adaptive gate, and use it to run a partially controlled ablation of frequency decomposition and spectral routing. We test this on five one-dimensional benchmark PDEs, ranging from smooth, single-scale problems to oscillatory, multi-scale ones. Frequency decomposition helps most on the spectrally complex benchmarks, cutting relative $L_2$ error by up to $59.2\%$ on a multimodal wave problem, but gives little benefit on smoother PDEs. On one benchmark (1D Wave), it performs substantially worse than a simpler fixed-combination variant. The gate's benefit scales with how spectrally rich the target solution is: the full model's advantage over the ablations is largest on multi-scale benchmarks and smallest (or negative) on single-scale ones, consistent with the gate exploiting frequency structure rather than acting as noise,though we do not directly visualize or quantify its spatial activations in this study. All results come from a single training seed across five 1D benchmarks, so we present this as an exploratory study meant to raise questions rather than answer them, and outline the additional seeds and benchmarks needed to test whether the pattern holds.

1 Introduction

PINNs can learn low-frequency solution components faster than high-frequency structure, motivating frequency-aware methods. This study tests when explicit frequency decomposition and adaptive spectral routing help across five one-dimensional PDE benchmarks.

  • Motivation: Spectral bias causes neural networks, including PINNs, to learn low-frequency target components before high-frequency ones.This makes localized discontinuities, sharp gradients, oscillations, and interacting frequencies difficult to approximate.
  • Motivation: Existing frequency-aware methods use richer inputs, periodic activations, or optimization changes, but often assume broad usefulness across PDE types.Examples include Fourier feature embeddings, SIREN activations, adaptive activations, loss balancing, and learning-rate annealing.
  • Research question: The paper asks whether explicit frequency decomposition consistently improves accuracy or instead depends on the PDE solution’s spectral character.The motivation is a direct comparison with matched-capacity ablations rather than an assumption that richer spectral representations always help.
  • Approach: DBSG-PINN separates low- and high-frequency components into subnetworks and combines them with an adaptive gate that learns spatially varying mixing weights.The low-frequency branch uses hyperbolic tangent activations, while the high-frequency branch uses sinusoidal activations.
  • Experimental design: The ablation framework removes the high-frequency branch, low-frequency branch, or gate while keeping capacity and optimization settings matched as closely as the architecture allows.The comparisons are designed to reduce, but not eliminate, confounding from model complexity or training strategy.
  • Findings: Frequency decomposition helps most on spectrally complex benchmarks, reaching up to 59.2% lower relative L2 error on Multimodal Wave, but offers little or negative benefit on smoother problems.A fixed-combination variant outperforms the full gated model on 1D Wave.
  • Scope: The gate’s contribution is largest on spectrally rich targets and smallest or negative on simple ones, although this benchmark-level pattern lacks direct spatial gate visualization.The study reports one trained model per benchmark across five benchmarks, so it is exploratory rather than a general claim about frequency-decomposed PINNs.

2 Methodology

The method formulates PINNs by minimizing PDE, initial-condition, and boundary-condition losses, then introduces DBSG-PINN to combine smooth and oscillatory subnetworks through a learned gate. Evaluation uses matched ablations and post-training accuracy, residual, and spectral diagnostics, with important controls and caveats.

  • Physics-Informed Neural Networks: PINNs minimize a weighted sum of PDE residual, initial-condition, and boundary-condition losses, with derivatives computed through automatic differentiation.The loss weights are fixed per benchmark so ablation variants share the same loss landscape.
  • Physics-Informed Neural Networks: For second-order-in-time wave benchmarks, the initial-condition loss includes both initial value and initial velocity; first-order benchmarks use only the initial value.Both wave initial-condition terms are obtained through automatic differentiation and weighted equally.
  • DBSG-PINN Architecture: DBSG-PINN uses low- and high-frequency branches intended for smooth and oscillatory components, trained jointly with an adaptive gate.The low-frequency branch uses tanh activations, while the high-frequency branch uses sinusoidal activations.
  • DBSG-PINN Architecture: “Frequency decomposition” denotes an activation-driven inductive bias rather than guaranteed spectral separation, because either branch can represent broader functions.The gate determines how much each branch contributes at each (x, t).
  • DBSG-PINN Architecture: The branches are combined as uθ(x, t) = g(x, t)uhi(x, t) + (1 − g(x, t))ulo(x, t), producing a convex combination of their predictions.The gate is a scalar sigmoid-style weighting mechanism for the two-branch mixture.
  • Ablation and Evaluation: The study compares the full model with NoGate, LowOnly, and HighOnly variants across five PDE benchmarks using approximately matched training settings and capacity.LowOnly and HighOnly are not clean like-for-like tests because branch depth and width differ, and evaluation also includes post-training finite-difference residuals.

3 Results

Across five one-dimensional PDE benchmarks, the ablation results compare DBSG-PINN components on solutions ranging from smooth single-scale to oscillatory multi-scale behavior. The reported outcomes indicate that decomposition is most useful for spectrally complex targets, while adaptive gating can hurt on a single-frequency wave benchmark.

  • Evaluation scope: All reported metrics come from a single training run per model and benchmark, so the results are exploratory observations rather than statistically confirmed effects.The study identifies relative L2 error, spectral error, and HF/LF recovery as evaluation metrics.
  • 3.1 1D Multimodal Wave Equation: The multimodal wave benchmark combines spatial modes k ∈ {1, 3, 5}, providing the study’s explicitly multi-scale target.Its governing equation is the one-dimensional wave equation with c = 1 and Dirichlet boundary conditions.
  • 3.2 1D Allen–Cahn Equation: Allen–Cahn uses a numerical reference solution because its benchmark has no simple closed-form solution.The reference is high-resolution and obtained independently of the trained network.
  • 3.3 1D Burgers Equation: The Burgers benchmark uses viscosity ν = 0.01/π and is evaluated against a high-resolution numerical reference rather than an analytical solution.It imposes Dirichlet boundary conditions u(−1, t) = 0 and u(1, t) = 0.
  • 3.4 1D Reaction–Diffusion Equation: The Reaction–Diffusion benchmark is a smooth, low-order case with a single spatial mode, sin(πx), whose amplitude changes over time.It uses the linear equation ut = D uxx + λu with D = 0.01 and λ = 2 on [0, 1] × [0, 1].

4 Discussion

Across benchmarks, the gate helps most when solutions contain richer spectral structure, but its advantage is neutral or negative on simpler targets and remains exploratory because results use one seed and one-dimensional PDEs.

  • Spectral complexity and decomposition: 59.2% lower relative L2 error occurs on Multimodal Wave, where the full model beats every ablation on nearly every metric.This target explicitly superposes multiple spatial modes, and each single-branch ablation performs worse at recovering the other branch’s intended spectral content.
  • Spectral complexity and decomposition: On Allen–Cahn, Full narrowly beats LowOnly, with relative L2 values of 0.1038 versus 0.1108.The smooth solution and localized transition layers make suppressing an unhelpful high-frequency branch more important than blending two useful branches.
  • Spectral complexity and decomposition: On Burgers, Full and NoGate remain nearly tied, with relative L2 values of 0.0238 and 0.0240, respectively.The viscosity-smoothed target is close to single-scale, leaving little spectral content for adaptive routing.
  • Spectral complexity and decomposition: On 1D Wave, NoGate leads Full on every tracked metric, by around 53% on relative L2 and 68% on spectral error.Because the target uses a single wavenumber, the authors did not expect much for the gate to route between and lack a confident explanation for the large gap.
  • Interpreting the gate: The gate’s aggregate benefit generally tracks spectral complexity, but Reaction–Diffusion is an unresolved exception despite having a single spatial mode.The interpretation relies on cross-benchmark performance comparisons rather than direct visualization of g(x, t).
  • Limitations: All results use a single training seed, roughly matched branch capacities, and five one-dimensional benchmarks, limiting claims about reproducibility and broader generalization.The authors specifically identify uncertain seed effects, unequal functional capacity, and unknown behavior in higher dimensions or under stronger nonlinearity.

5 Conclusion

The study finds that frequency decomposition helps most on spectrally complex PDEs, while simpler variants can perform better on smoother benchmarks. DBSG-PINN provides a testable pattern, but the single-seed, five-benchmark design requires broader validation.

  • Up to 59.2% lower relative L2 error was achieved on Multimodal Wave, whose solution is a superposition of spatial modes.This was the benchmark where decomposition helped most.
  • The Full model led on Multimodal Wave, Allen–Cahn, and Reaction–Diffusion, whereas NoGate led on Burgers and 1D Wave.
  • On 1D Wave, NoGate beat the Full model by a wide margin on every tracked metric, while Burgers showed no meaningful learned-gating advantage over a fixed average.
  • The gate’s benefit tracked benchmark spectral complexity, but this remains a preliminary interpretable signal rather than a proven mechanism.
  • The study’s concrete pattern is that decomposition helps most on genuinely multi-scale solutions and least on smooth, single-scale ones.
  • Multiple seeds and a wider benchmark suite are needed before drawing firmer conclusions for PINN practice.

A.1 Hyperparameter Configuration

The appendix documents the shared DBSG-PINN configurations and implementation choices used across the five benchmark PDEs.

  • Table 6 lists branch architectures, gate configuration, training settings, loss weights, and the random seed for all five benchmark PDEs.
  • All entries use a single random seed per benchmark, limiting statistical confidence in the reported configurations.
  • The models were implemented in Julia with NeuralPDE.jl, Lux.jl version 1.2.3, and grid-based collocation.

A.2 1D Multimodal Wave Equation: Qualitative Comparison Across Variants

Figures 3–7 compare the Multimodal Wave reference solution, predictions, absolute errors, and, where shown, FFT-based spectral recovery across ablation variants.

  • Figures 3–7 present exact solution, prediction, pointwise absolute error, and where shown FFT-based spectral recovery for each ablation variant.
  • Consistent color scales within each figure support direct visual comparison across its panels.
  • Each figure corresponds to a single trained model.

A.3 1D Burgers Equation: Qualitative Comparison Across Variants

Figures 8–12 provide matched qualitative comparisons for the 1D Burgers benchmark, using a high-resolution numerical solution as the reference.

  • Figures 8–12 compare the Burgers reference solution, predictions, pointwise absolute errors, and spectral recovery across ablation variants.
  • The Burgers reference is a high-resolution numerical solution rather than a closed-form exact solution.

A.4 1D Allen–Cahn Equation: Qualitative Comparison Across Variants

Figures 13–17 present the qualitative comparison across variants for the 1D Allen–Cahn benchmark.

  • A.4 1D Allen–Cahn Equation: Qualitative Comparison Across Variants: Figures 13–17 cover the 1D Allen–Cahn benchmark comparison across variants.
  • A.4 1D Allen–Cahn Equation: Qualitative Comparison Across Variants: The Allen–Cahn comparison is presented as the same comparison used for the surrounding benchmark analyses.
  • A.4 1D Allen–Cahn Equation: Qualitative Comparison Across Variants: The cited passage identifies Figures 13–17 as the visual material for this benchmark’s qualitative variant comparison.

A.5 1D Reaction–Diffusion Equation: Qualitative Comparison Across Variants

The supplied figures organize qualitative and spectral comparisons across multimodal wave, Burgers, Allen–Cahn, reaction–diffusion, and wave benchmarks.

  • 1D Multimodal Wave: Figures 5–7 show the 1D Multimodal Wave LowOnly and NoGate variants alongside spectral recovery across ablations.
  • 1D Burgers Equation: Figures 8–12 show the 1D Burgers benchmark’s Full DBSG-PINN, HighOnly, LowOnly, NoGate, and spectral-recovery views.
  • 1D Allen–Cahn Equation: Figures 13–17 provide the corresponding 1D Allen–Cahn comparison, including Full DBSG-PINN, HighOnly, LowOnly, NoGate, and spectral recovery.
  • 1D Reaction–Diffusion Equation: Figures 18–22 provide the corresponding 1D Reaction–Diffusion comparison, including exact solution, prediction, absolute error, variants, and spectral recovery.
  • 1D Wave Equation: Figures 23–27 provide the corresponding 1D Wave comparison, including exact solution, prediction, absolute error, variants, and spectral recovery.
Loading 2608.24940v1…