PaperlayerView exact revision on arXiv

Source-linked AI summary

When does least-squares residual minimization solve a differential equation?

Carlos Esteve-Yagüe, Richard Tsai, Stanley Osher

arXiv:2609.35716v1math.NAcs.NA

TL;DR

The paper asks when residual minimization and its gradients justify concluding that a trial function solves a differential equation. It analyzes these implications across several equation classes and identifies both failures and conditions that restore them. Its results show that stationarity, sampled gradients, and minimizing sequences require distinct justification.

  • Problem

    It is unclear when stationarity, sampled gradients, or minimizing sequences of a residual loss justify concluding that a trial function solves the intended differential equation.

    Source: §1 · p. 2, Relation to existing work and… · p. 5

  • Method

    The paper analyzes residual orthogonality, admissible and parameterized variations, gradient consistency, and minimizing-sequence limits across several equation classes.

    Source: Relation to existing work and… · p. 5, §2.1 · p. 8

  • Results

    The implications can fail: transonic conservation-law minimizing sequences approach a continuous non-solution, while moving residual jumps can make sampled gradients omit terms in the continuous gradient.

    Source: §7 · p. 39

  • Takeaways & Limitations

    Stationarity, gradient consistency, and convergence to an admissible solution are separate implications, each requiring its own supporting conditions.

    Source: Relation to existing work and… · p. 5, §7 · p. 39

    For finite-dimensional parameterizations with residual space L2, the dense-image condition needed for the stated critical-point result cannot hold.

    Source: §2.2 · p. 11

Abstract

from arXiv · show

Least-squares residual minimization approximates a differential equation through an optimization problem. This paper investigates the conditions required to transition from stationary points and minimizing sequences of the residual functional to the exact solution. Starting from the classical orthogonality relation between the residual and the range of the linearized operator, we study how boundary conditions and parameterized variations determine what can be inferred about the residual at a stationary point. Smooth spurious critical points can occur even for well-posed problems; for linear equations, a compatibility condition yields an exact projection characterization. For a scalar conservation law with transonic boundary data, every minimizing sequence converges to a continuous function that does not solve the equation, even when a stationary entropy shock exists. We give conditions under which approaching this limit forces the parameter norms to diverge. A Hamilton--Jacobi example shows that a function can have zero residual without being the viscosity solution. Favorable results for a viscous model and uniformly convex Monge--Amp\`ere critical points identify conditions that restore the corresponding implications. For piecewise-smooth trial families, the regularity needed to define the residual functional does not by itself justify the sampled gradient used in training. Moving residual jumps can contribute terms absent from that gradient; an explicit example has identically vanishing sampled gradients and a nonzero continuous derivative. Together, these results clarify what stationarity and sampled gradients imply about the residual, and under which conditions minimizing sequences lead to the intended solution.

1. Introduction

The introduction distinguishes stationarity, sampled-gradient consistency, and minimizing-sequence convergence as separate steps between residual optimization and solving a differential equation. It motivates these questions with counterexamples and previews conditions that support the desired implications.

Two motivating examples.

  • The well-posed Neumann problem has a smooth non-solution critical point with positive loss, showing that well-posedness alone does not ensure every critical point solves the equation.At u = 0, the residual is orthogonal to the range of the linearized operator because the Neumann Laplacian has zero-mean range.Source: Two motivating examples · p. 3
  • For the Dirichlet–Poisson problem, full admissible variations force a critical point to solve the equation, yet sampled-gradient training can reach positive-loss plateaus.The figure reports an independently evaluated integral loss of approximately 0.241 for the illustrated network run.Source: Two motivating examples · p. 4, Figure 1 · p. 4

Relation to existing work and contributions.

  • The paper treats stationarity, gradient consistency, and convergence of minimizing sequences as distinct questions requiring separate arguments.Resolving one does not settle the others: more accurate sampling, approximation capacity, or an exact continuous gradient alone is insufficient.Source: Relation to existing work and… · pp. 5–6
  • For linear equations, a compatibility condition on the trial parameterization yields an exact projection characterization of critical residuals.The characterization makes explicit what stationarity implies about the residual.Source: Relation to existing work and… · p. 5
  • For scalar conservation laws with strictly convex flux and transonic boundary data, every minimizing sequence approaches the same continuous profile outside H1 that is not a solution, even when an entropy-admissible stationary shock exists.The infimum is positive and unattained; under the stated derivative-bound condition, approaching it forces parameter escape.Source: Relation to existing work and… · pp. 5–6
  • For piecewise-smooth trial families, moving residual jumps contribute terms absent from the sampled gradient, and an explicit Poisson example has identically zero sampled gradients but a nonzero continuous derivative.The resulting bias is independent of sample size; continuity of the residual removes it under the stated assumptions.Source: Relation to existing work and… · p. 6

2. Residual calculus and parameterized trial functions

The section relates residual minimization to the range of the linearized residual map, then examines how boundary conditions, parameterized variations, and sampling constrain what stationarity implies. It distinguishes continuous first variations from sampled gradients and identifies conditions under which they agree.

2.1. Residual functions, residual maps, and residual spaces.

  • A critical point has residual orthogonal to the linearized residual range; dense range or a trivial adjoint kernel therefore forces zero residual.A nonzero critical residual must lie in the adjoint kernel, but kernel elements are not necessarily realized as critical residuals.Source: §2.1 · p. 9
  • With penalized boundary conditions, stationarity makes the interior and boundary pairings sum to zero, but does not force either pairing to vanish separately.The residual and boundary operator are treated together in a weighted product space.Source: §2.1 · p. 9

2.2. Parameterized functions and admissible variations.

  • At a parameter critical point, the residual is orthogonal to the linearized residuals generated by admissible parameter variations.Thus stationarity depends on the image of the restricted linearized map, not simply on the full variation space.Source: §2.2 · pp. 10, 12
  • The dense-range criterion cannot hold for finitely many parameters with the standard infinite-dimensional residual space L2(Ω).The criterion applies only with infinitely many parameters or a finite-dimensional residual space.Source: §2.2 · p. 11
  • Boundary conditions can block the full-space range condition: in the Neumann example, the residual at u = 0 is orthogonal to every linearized variation.The Neumann constraint makes each function in the linearized range have mean zero, while the residual is the constant −1.Source: §2.2 · p. 11
  • For nonlinear residual maps, the second variation has a sign-indefinite term, so the least-squares functional can be nonconvex.The range arguments rely only on the first-order identity and the geometry of the restricted linearized map.Source: §2.2 · pp. 11–12

2.3. Sampled gradient.

  • Under the stated integrability and differentiability conditions, sampled losses converge to the continuous loss and sampled gradients converge to the mean sampled gradient, which equals the continuous gradient when the residual map is Fréchet differentiable.The loss consistency result requires a square-integrable residual; identifying the mean gradient with the continuous gradient requires the additional differentiability condition.Source: §2.3 · p. 13
  • Parameter-dependent residual jumps can add interface-motion terms to the continuous derivative that the sampled gradient omits, even when the sampled loss is consistent.A deterministic quadrature rule converging for the pointwise residual-gradient integrand can share the discrepancy.Source: §2.3 · pp. 13–14

3. Linear equations: projection geometry

For linear equations, criticality makes the residual orthogonal to the image of the admissible parameter variations, yielding an exact projection characterization under a compatibility condition. A stability estimate can turn the projection distance into an error bound, but a small parameter gradient alone is insufficient without control of conditioning.

3. Linear equations: projection geometry

  • At a critical point, the residual is the orthogonal component of the boundary-shifted data relative to the image of admissible variations, under the stated compatibility condition.This gives an exact projection characterization of critical residuals for linear equations.Source: §3 · pp. 14–15
  • The exactness criterion is Auθ∗ = f if and only if f − Aub lies in the image of the admissible variation space.A positive residual therefore records data not represented by that image.Source: §3 · p. 15
  • A usable stability estimate converts the projection distance into an a posteriori solution-error bound, and the distance is computable after training.The bound vanishes exactly when the shifted data lie in the image of the admissible variations.Source: §3 · p. 15
  • A small Euclidean parameter gradient alone does not ensure a small solution error unless the conditioning of the nonzero directions Awk is controlled.The residual's component in the image is reflected in the parameter gradient, while the perpendicular component can remain.Source: §3 · p. 16
  • For a linear trial space, the image of the variations is fixed, so the problem reduces to convex quadratic least squares and ordinary projection onto that image.Neural parameterizations generally have parameter-dependent variation spaces; the trial functions must also be regular enough for the strong residual.Source: §3 · pp. 16–17

4. Scalar conservation laws

This section examines when residual minimization for scalar conservation laws yields the intended solution, contrasting the effects of positive viscosity and transonic inviscid minimization. It also considers positive-loss plateaus and parameter growth, including cases where analytic iterates closely approximate the target profile.

4.1. The effect of viscosity.

  • For positive viscosity, every stationary trial function satisfying the Dirichlet data has zero residual, and every minimizing sequence converges strongly in H2 to the viscous solution.For fixed positive viscosity, the residual controls the trial functions; the estimates do not imply bounded neural-network parameters.Source: §4.1 · pp. 18–20
  • In the inviscid nonsonic case, stationarity forces the residual to be constant, but a constant residual need not vanish; additional residual regularity can relax nonsonicity.For an H1 residual, stationarity gives f′(u)r_x = 0, so a nonzero f′ almost everywhere makes r constant.Source: §4.1 · p. 19

4.2. Residual minimization and stationary entropy solutions.

  • For transonic inviscid data, every minimizing sequence converges uniformly to a continuous, non-H1 limit with nonzero residual, not to an entropy-admissible stationary shock when one exists.The infimum is positive and unattained; residuals converge strongly in L2 to a nonzero function.Source: §4.2 · pp. 21–23
  • When parameter bounds control derivative norms, convergence toward the transonic minimizing limit forces parameter norms to diverge.The limit lies outside H1, while minimizing sequences have derivative norms tending to infinity; the theorem transfers this escape to the parameterized family.Source: §4.2 · pp. 22, 24
  • For reversed transonic data, minimizing sequences likewise converge to a continuous cusp outside H1, and the limit is not a stationary weak solution.The same parameter-escape conclusion holds under the bounded-parameter assumption.Source: §4.2 · p. 24
  • 0.503 was the inviscid Burgers loss, approaching the predicted infimum 1/2; the viscous comparison reached 2.44 × 10^-8.The inviscid trial approached the predicted limit with a narrow residual transition, while the viscous run's inner weights and derivative maxima stabilized.Source: §4.2 · p. 25, Figure 3 · p. 26
  • The loss can plateau at a strictly positive value as parameters grow, although every finite iterate remains analytic and closely approximates u†; it controls (f(uθ))′, not derivative norms where regularity changes.Source: §4.2 · p. 26

5. Fully nonlinear equations

The fully nonlinear examples show that stationarity and residual size identify the intended solution only under additional conditions. Uniform convexity makes Monge–Ampère critical points solve the equation, while Hamilton–Jacobi functions can have zero residual without being viscosity solutions.

5.1. The Monge–Amp`ere equation.

  • Uniformly convex C3 Monge–Ampère critical points solve the equation because the linearized Dirichlet test problem is elliptic and the first-variation identity forces r = 0.The argument needs only r ∈ L2 and does not differentiate the residual.Source: §5 · p. 27, §5.1 · pp. 28–29
  • Without uniform convexity, the cofactor matrix can degenerate, and a convex but not uniformly convex function can be a spurious critical point with J(0) = π/2 > 0.If convexity is also dropped, the cofactor matrix may be indefinite.Source: §5.1 · p. 29
  • The function-space Monge–Ampère result does not transfer automatically to finite-dimensional parameterizations, whose variation images cannot be dense in L2 for a fixed finite model.Whether a parameterized or collocation discretization inherits the property is a separate approximation and rank question.Source: §5.1 · p. 29

5.2. Hamilton–Jacobi equations: zero residual without viscosity selection.

  • For the eikonal problem, a continuum of sawtooth profiles has zero residual, although the viscosity solution is the distance function min(x, 1−x).Stationarity permits both upward and downward corners; the viscosity condition rejects the local minimum corner.Source: §5 · p. 27, §5.2 · pp. 29–30
  • In this eikonal example, semiconcavity excludes upward slope jumps and, with zero endpoint data, selects the distance profile; more generally, zero-residual functions need an additional admissibility criterion.Suggested approaches include an appropriate constraint, vanishing viscosity, or a consistent monotone discretization with the required stability, comparison, and residual estimates.Source: §5.2 · p. 30

6. Minimal residual regularity and consistency of sampled gradients

The regularity that makes a strong-residual loss well defined does not ensure that sampled gradients match its continuous first variation. For piecewise-smooth trial families, moving residual jumps add interface terms, so stationarity and training gradients require conditions beyond residual definability.

6.1. Cellwise adjoints and interface conditions.

  • For variations on a fixed partition, criticality is equivalent to cellwise adjoint equations plus boundary and interface conditions.The interface conditions include [[ar]] = 0 and [[br − (ar)x]] = 0 at interior interfaces.Source: §6.1 · p. 32
  • Finitely many parameter directions need not impose the cellwise adjoint equation or its boundary and interface conditions, so stationarity alone need not force zero residual.Concluding that the residual vanishes requires adjoint uniqueness with those conditions or a separate dense-range argument.Source: §6.1 · p. 32

6.2. Parameter-dependent interfaces and bias in the sampled gradient.

  • Moving residual jumps add an interface contribution to the continuous gradient that is absent from the mean sampled gradient, independently of sample size; continuity of the residual removes this bias.The loss estimator remains unbiased, and the gradient bias vanishes precisely when interface contributions cancel.Source: §6.2 · p. 34
  • A squared-ReLU Poisson example has zero sampled derivative almost surely at fixed sample points but a nonzero continuous derivative, while its sampled loss converges to the continuous loss.The parameterized trial map is not C1 into H2 ∩ H1_0, so the stated hypotheses for the earlier results do not apply.Source: §6.2 · p. 35
  • For parameter variations, volume and trace contributions can cancel; interface motion is represented through sensitivities and traces without requiring K = 0.The residual-level identity ∇θJ = G + K also applies to higher-order equations under its stated hypotheses.Source: §6.2 · p. 36

6.3. Numerical comparison for the Poisson equation.

  • Adding the interface correction lowered terminal loss in every seed by factors of 4.0–49 with quadrature and 3.9–34 with 100 samples; quadrature also lowered loss in every seed.The terminal losses were evaluated by the same cellwise quadrature across all four variants.Source: Table 1 · p. 37, §6.3 · p. 37
  • With 100 samples, |GQ| ranged from 1.9 to 5.9, |K| from 4.8 to 13, and gradient fluctuation from 8.1 to 15.5, or 0.71–1.60 times |GQ + K|.Sampling fluctuations therefore remained comparable to the continuous gradient despite the larger sample count.Source: §6.3 · p. 38

7. Conclusion

The paper separates what stationarity, sampled gradients, and minimizing sequences establish about a differential equation’s solution. Its examples show both failures without additional conditions and cases where such conditions restore the desired implications.

7. Conclusion

  • Stationarity gives residual orthogonality to the linearized operator’s admissible variations, while boundary conditions and parameterization restrict which variations are tested.A minimizing sequence additionally requires convergence within the admissible class and a limit that retains the required regularity and intended-solution properties.Source: §7 · pp. 38–39
  • In the transonic model, every minimizing sequence converges uniformly to a continuous function outside H1 that is not a weak solution, with diverging derivative L2 norms.If bounded parameters imply bounded derivative L2 norms, approaching the infimum requires parameter norms to diverge.Source: §7 · p. 39
  • Moving residual jumps can contribute to the continuous gradient independently of sample size, while continuity of the residual removes this bias under the stated assumptions.The paper also identifies favorable conditions for linear equations, a viscous model, and uniformly convex Monge–Ampère critical points.Source: §7 · p. 39
  • A small loss gradient must be interpreted through the admissible variations and their conditioning; accurate sampled losses alone do not establish gradient consistency or residual control.An error estimate also requires residual stability and conditions selecting the intended solution; more points or model capacity alone do not provide these properties.Source: §7 · p. 39
Loading 2609.35716v1…