PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry-Constrained Kolmogorov-Arnold Networks: Learning Edge Geometry via Banach Duality

K S Sesh Kumar

arXiv:2608.25807v1cs.LGstat.ML

TL;DR

KANs rely on learnable edge functions, but fixed bases impose function-space geometry before data are observed. The paper introduces geometry-constrained KANs that learn this geometry through per-edge exponents, matching or improving fixed-basis baselines with especially strong noise robustness and small-sample performance.

  • Problem

    Fixed-basis KANs impose function-space geometry before observing data, while discontinuous targets can expose mismatches between that geometry and target structure.

  • Method

    Geometry-constrained KANs derive edge activations from duality maps and learn an exponent p per edge to adapt the underlying geometry.

  • Results

    Across the benchmark, geometry-constrained KANs match or improve fixed-basis baselines, with ℓp-KAN degrading 3.7× under noise versus 21.6× for unregularised splines.

  • Takeaways & Limitations

    Learned geometry provides an interpretable, init-anchored signal and is particularly effective under measurement noise and in the small-sample regime.

  • Takeaways & Limitations

    The trainable-depth prediction p > 3/2 is not directly tested in deep stacks, and the experiments use shallow symbolic-regression models.

Abstract

from arXiv · show

Kolmogorov-Arnold Networks (KANs) replace fixed activations in deep architectures with learnable univariate edge functions, making the choice of edge parametrisation central. Existing variants rely on fixed bases such as splines, polynomials, or Fourier features, which impose a function-space geometry before data are observed. We introduce geometry-constrained KANs, a family of edge activations derived from Banach duality maps in which the geometry itself is learned through a scalar exponent $p > 1$ per edge. This exponent controls the qualitative response: sub-Euclidean values produce sharp, threshold-like behaviour reminiscent of the $\ell_1$ (LASSO) geometry, $p = 2$ recovers the linear regime, and larger values produce flatter responses near the origin. Across 50 symbolic-regression targets ($40$ from the AI Feynman benchmark plus $10$ synthetic stress tests), geometry-constrained KANs match or beat every fixed-basis baseline on median NRMSE (Banach-KAN $0.030$, tying Chebyshev and improving on splines); on average rank Banach-KAN is best on the $18$-equation core ($2.00$) and statistically tied with the strongest spline on the full benchmark ($2.32$ vs. $2.34$). The clearest gains appear under measurement noise: as $σ$ grows from $0$ to $1$, $\ell^p$-KAN degrades only $3.7\times$ -- below even a cross-validated spline ($\approx 11\times$) -- while an unregularised spline degrades $21.6\times$; Banach-KAN degrades $8.8\times$, comparable to a tuned spline but far more stable than the unregularised one. Banach-KAN also takes the most per-equation wins in the small-sample regime, with fixed-basis models catching up only as the training set grows. Learned exponents provide an interpretable, relative signal: at a fixed initialisation they reveal a consistent, target-dependent geometric ordering across equation families and input dimensions.

1 Introduction

KANs make learnable edge functions central, but fixed-basis variants impose geometry before seeing data and can mismatch target structure. Geometry-constrained KANs instead learn an interpretable per-edge exponent, with empirical gains in robustness and small-sample regression.

  • KANs replace fixed neural activations with learnable univariate edge functions, making edge-function choice the central modelling decision.
  • Fixed bases impose function-space geometry before data are observed, encoding inductive biases such as smoothness, periodicity, or locality.
  • Matched-budget fits show no fixed basis performs uniformly well across discontinuous targets because geometric mismatch produces oscillations or bias.Figure 1 uses 21 trainable parameters, N = 2000 samples, and label noise σ = 0.03.
  • Geometry-constrained KANs learn the geometry through a per-edge exponent p and move edges between sharp, linear, and saturating regimes.The approach parameterises each edge with a duality map rather than selecting a fixed basis.
  • Banach-KAN provides the best average rank on the 18-equation core and learned exponents give an interpretable, init-anchored signal of task structure.The contribution also reports better small-sample performance than fixed bases and explicit spline regularisation.

2 Background and Geometry

KANs can be understood through the geometry induced by their edge functions. The paper replaces fixed geometry with learnable duality-map parameters, yielding activations whose shape adapts continuously to data.

  • KANs aggregate learnable univariate edge functions, so the model class is determined by the nonlinear transformations assigned to edges.
  • Existing basis expansions fix the function space before observing data, imposing inductive biases that determine how functions are represented.Examples include B-splines, radial basis functions, wavelets, Chebyshev and Jacobi polynomials, and Fourier features.
  • Fixed geometries produce distinct failure modes on discontinuous targets: smooth bases distribute discontinuities across coefficients, causing oscillations or bias.
  • Adaptive-activation methods learn input slope, whereas geometry-constrained KANs learn the exponent p and thereby change the shape and regularity class.The reported regimes are threshold-like as p →1, linear at p = 2, and flattened for large p.
  • Convex potentials define geometries, their gradients define activations, and unit balls visualise how smaller p sharpens responses while larger p flattens origins and amplifies tails.
  • Learning p corresponds to learning the underlying geometry, providing continuous per-edge adaptation without discrete basis selection or architectural search.

3 Geometry-Constrained KANs

Geometry-constrained KANs replace fixed edge bases with activations derived from geometric duality maps, learning an exponent p per edge. The resulting family spans sharp, linear, saturating, and combined response regimes while yielding a trainable-depth prediction for ℓp edges.

  • Family definition: Geometry-constrained KANs use parameterized univariate edge activations with affine pre-activations, forming a controlled family of geometry–expressivity trade-offs.The tanh edge has four learnable parameters per edge, while the geometry-constrained variants add geometry-specific parameters.
  • ℓp geometry: The learned exponent p continuously controls edge shape, producing sharp threshold-like responses near p → 1, approximately linear behavior near p = 2, and flatter near-origin responses for larger p.This makes p a compact descriptor of local regularity rather than a fixed basis or separately tuned regularization parameter.
  • Family definition: Banach-KAN combines tanh and ℓp duality-map branches, allowing bounded and power-law responses that neither branch achieves alone.Independent affine parameters let the two geometries attend to different input projections, trading exact shared-affine convex structure for expressivity.
  • What the geometry predicts: trainable depth: The trainable-depth condition for the exact ℓp duality map is p > 3/2: only then is the Jacobian order parameter finite and a critical weight scale exists.At or below p = 3/2, the relevant Gaussian moment diverges for every variance, preventing a critical scale in the plain feedforward setting.
  • What the geometry predicts: trainable depth: For the implemented shallow symbolic-regression models, only 1.2–5.4% of edges are sub-3/2, while the direct deep-stack test of the prediction remains future work.The regularized implementation keeps the order parameter finite for all p > 1, but the critical scale vanishes as ε → 0 below p = 3/2.

4 Practical Implementation and Inference

Training jointly estimates each edge’s exponent and standard activation parameters, using unconstrained optimization with p = 1 + exp(θ). An everywhere-C1 map and gradient clipping stabilize sub-Euclidean edges while reproducing the additive-ε results.

  • Optimization: The fitting procedure estimates the per-edge exponent together with standard activation parameters from training data.The exponent is optimized through θ, while the remaining edge parameters are collected in Θ.
  • Optimization: The parameterization p = 1 + exp(θ) enforces p > 1 while allowing unconstrained optimization with Adam, cosine annealing, and gradient clipping at 10.The optimization runs for 5,000 steps, with θ as the only non-standard parameter.
  • Numerical stabilization: The adopted everywhere-C1 duality map removes the origin’s derivative singularity for p < 2 and replaces it with a bounded smooth peak controlled by ε.Experiments use ε = 10^-8; the C1 and additive-ε curves agree outside an ε-neighborhood of zero.
  • Numerical stabilization: Gradient clipping controls near-origin sensitivity, where the smoothed slope can reach approximately 10^4 for (p, ε) = (1.5, 10^-8).The clip therefore fires routinely on sub-Euclidean edges and keeps training stable.
  • Numerical stabilization: The C1 correction reproduces additive-ε performance: clean medians agree within 0.5% and noise-degradation ratios are unchanged.Thus, the differentiability correction changes no result-table entry.

5 Experiments

Experiments evaluate geometry-constrained KANs on benchmark equations, noise, sample-size, real-world regression, extrapolation, and scope probes. Across these settings, adaptive geometry is strongest or especially robust under noise and limited data, while its advantages remain bounded by the studied regime.

  • Protocol: The evaluation uses 50 symbolic-regression targets, PMLB as a tabular sanity check, shared splits, input normalisation, and median NRMSE.The benchmark contains 40 AI Feynman equations and 10 synthetic stress tests; the Core 18 subset supports noise and small-sample sweeps.
  • Benchmark equations: Banach-KAN achieves the best average rank on the full 50-equation set and Core 18, at 2.32 and 2.00 respectively.Spline G=3 takes the most individual wins but has a bimodal error profile, whereas Banach-KAN is uniformly competent.
  • Learned geometry: Across 4,465 Banach-KAN edges, 18.7% learn p < 2 and the sub-Euclidean fraction rises from 11% at 2D to 34% at 6D.The ordering is target-dependent and stable under noise, but the authors qualify it as a relative, aggregate, initialisation-anchored signal rather than an absolute per-edge threshold.
  • Noise robustness: 3.7× and 3.2× are the noise degradations of ℓp-KAN and Tanh-KAN, versus approximately 11× for cross-validated splines and 21.6× for Spline G=3.Banach-KAN degrades 8.8×; it is strongest at low noise, while ℓp-KAN and Tanh-KAN overtake it at σ = 1.0.
  • Small-sample approximation: 8/18 and 7/18 are Banach-KAN’s wins at n = 50 and n = 100, the most of any single model in the small-sample sweep.Across 90 sample-size conditions, Banach-KAN wins 28, and it is the only family member described as uniformly competitive across the budget range.
  • Extrapolation: 0.35 and 0.42 are Banach-KAN’s and ℓp-KAN’s median NRMSE values under out-of-range testing, ahead of Spline at 0.73 and Cheby at 0.80.The protocol trains on the inner 60% of each input box; the bounded Tanh edge is worst among geometry variants at 1.10, consistent with saturation.
  • Scaling and scope: The study is scoped to low-to-mid-dimensional regression, and the reported ResNet readout probe is not a depth test because the faithful inner-layer variant did not complete.The authors flag the growing Lipschitz constant of the unbounded power-law readout as a hypothesis about the limiting factor.
  • Tabular sanity check: 10/12 PMLB datasets are won by the geometry-constrained family, while Spline G=3 wins zero.Tanh-KAN is the strongest single model with 5 wins and mean rank 1.67; ℓp-KAN wins 4 datasets with power-law or heavy-tailed structure.

6 Discussion and Conclusion

The discussion frames learned function-space geometry as the central design choice in KANs and reports benefits at matched per-edge budgets, under noise, and with small samples. It also bounds the claims to specific operating regimes and identifies capacity and evaluation-scope limitations.

  • Banach-KAN achieves the best average rank on the Core 18 (2.00) and ties the strongest spline on all 50 equations (2.32 vs. 2.34).It matches or beats fixed-basis baselines on median NRMSE.
  • Learned geometry is presented as an interpretable design principle whose empirical effects differ by noise and sample-size regime rather than as a universal state-of-the-art claim.The paper positions the result at comparable per-edge budget, under measurement noise, and at small n.
  • 3.7× and 3.2× degradation for ℓp-KAN and Tanh-KAN remain below cross-validated spline degradation (≈11×), while Spline G=3 degrades 21.6×.Banach-KAN degrades 8.8× from σ = 0 to σ = 1.
  • Banach-KAN takes the most small-sample wins, while Spline G=3 gains wins as n increases and larger spline grids win rarely in the tested range.The comparison uses matched per-edge budgets rather than a parameter-efficiency claim over splines.
  • The approach has a capacity ceiling that fixed-basis KANs close as n grows, and evaluation is limited to symbolic regression and tabular data.Proposed future work adds learnable basis components, deeper compositions, and higher-dimensional modalities.

D Banach a2 = 0 ablation

The ablations show that Banach-KAN’s duality-map branch contributes materially beyond its tanh component, while learned exponents generally outperform fixed choices across clean and noisy equations. Noise robustness and small-sample performance vary systematically across models.

  • 1.91× median degradation follows from disabling the Jp branch, while full Banach-KAN beats the tanh-only variant on all 18 equations by at least 10%.The largest single-equation gap is 5.00× on trigonometric equation III.17.37.
  • 15/18 clean and 14/18 noisy equations favor learned-p Banach over fixed-p = 2 Banach, with only marginal fixed-p wins.The fixed wins are Capacitor, Doppler, and Barometric.
  • In the clean 18-equation benchmark, Spline G=3 wins 10/18 equations and Banach-KAN wins 8/18, while Banach remains strongest by average rank.The spline’s wins are localized; Banach is described as uniformly competent.
  • 3–4× degradation occurs for ℓp-KAN and Tanh-KAN, versus 21.6× for Spline G=3, as noise increases from σ = 0 to σ = 1.0.All spline variants reach NRMSE around 0.75–0.80 at σ = 1.0.
  • Banach-KAN leads at every n ≤500 and remains near the front at n = 1000, while Spline G=3 increases its wins from 3 to 7.The pattern is consistent with basis-based KANs needing more samples to identify per-edge parameters.

G Architecture dissection per equation

The architecture dissection finds a regime-dependent division of labor among the three geometry-constrained variants: Banach dominates clean data, whereas ℓp and Tanh gain wins under noise.

  • 18/18 clean equations favor Banach-KAN, while at σ = 0.3 Banach wins 10, ℓp wins 6, and Tanh wins 2.The comparison uses n = 500 on the 18-equation core.
  • Power-law geometry is described as most useful when label information is weak, while the bounded tanh branch is most useful for tracking fine compositional structure.

H Learned geometry: full distributional analysis

The learned exponents specialize across edges and organize by equation type and input dimension, providing an interpretable relative signal that remains stable under noise while individual values depend on initialization.

  • The fraction of edges with p < 2 increases monotonically with input dimension across 4,465 Banach-KAN edges.The distribution is summarized by input dimension in Figure 5 and Table 25.
  • 77% of training runs have within-run learned-p standard deviation above 0.5, indicating distinct edge geometries rather than collapse to one shared value.
  • 31% of hyperbolic, 28% of logarithmic, and 26% of trigonometric equations push edges toward p < 2, versus 11% of polynomial and 9% of power equations.
  • Mean learned p remains stable as σ grows, with only a slight widening of its across-equation standard deviation.
  • 17/18 clean equations favor learned p over fixed p in ℓp-KAN, while fixed p = 2 is consistently the worst choice.The sole fixed-p win uses p = 3.0 and is within 5% of learned p.

I Pendulum interpretability case study

The pendulum case study tests interpolation and extrapolation near a logarithmic singularity, showing that geometry-adaptive edges can preserve the target's local shape beyond the training interval.

  • Target geometry: The pendulum correction R(θ0) is smooth and monotone on the training domain but has a logarithmic singularity as θ0 approaches π.Near the singularity, derivatives grow with inverse powers of the distance to π and logarithmic factors.
  • Extrapolation: The Jp-stack extrapolates with the correct power-law shape, whereas the spline becomes flat beyond the training interval.The comparison uses parameter-matched pedagogical implementations, so its absolute legend NRMSE values exceed the full-architecture table values.
  • Interpolation: NRMSE 0.0012 at n = 500: ℓp-KAN dominates 1D interpolation across every sample size.This corresponds to a relative error below 0.2%.
  • Extrapolation: Banach-KAN dominates extrapolation after training only on [0.01, 2.0], transferring best toward the unseen near-singularity region.Tanh and MLP exceed NRMSE 1, while Spline reaches approximately 0.7.
  • Interpretability: Learned pendulum exponents average p = 2.9 ± 0.1 across five seeds, with individual edges spanning 2.3 to 4.9.The paper relates this value to the target's regularity near the upper training boundary, while framing a precise approximation-theoretic derivation as open.

J PMLB tabular benchmark: additional notes

Additional PMLB observations show that learned exponents vary with target smoothness, while model advantages depend on whether tabular structure is weak or scientifically structured.

  • Learned exponents: Learned p clusters around 2.5–3.0 across PMLB datasets, with an environmental dataset reaching approximately p ≈3.9.The higher value is described as consistent with smoother target functions favouring higher-order polynomial growth.
  • Model comparison: Tanh-KAN dominates tabular data, while ℓp and hybrid geometries show their largest advantages in structured scientific settings.The passage attributes the tabular pattern to weak compositional structure and sufficient Orlicz geometry.

K Memorisation and edge-expressivity toy experiments

Toy memorisation and edge-expressivity experiments indicate that performance depends on target geometry, with Banach-KAN broadly strong but specialised branches useful for saturation and discontinuities.

  • Memorisation tasks: No universal winner appears across the four toy tasks; model choice depends on smoothness and dimensionality.The tasks include exponential-trigonometric, Bessel, multiplication, and four-dimensional compositional targets.
  • Memorisation tasks: Banach-KAN is the best all-rounder at moderate-to-large data sizes, while Tanh-KAN outperforms Spline on noisy 2D and 3D targets.These patterns are reported across the toy tasks as consistent with the broader Feynman results.
  • Edge expressivity: NRMSE < 0.04 on each univariate target: Banach-KAN reproduces sigmoid, cubic, scaled Bessel J0, and exponential functions.ℓp-KAN succeeds on the two power-law-shaped targets but underperforms on the bounded sigmoid.
  • Edge expressivity: The tanh branch is needed when the target saturates, providing a geometric explanation for the conjunction architecture.The ablation connects bounded sigmoid behaviour with the specialised role of the tanh component.
  • Budget sensitivity: The geometry-adaptive edge remains at the noise floor across parameter budgets on the Heaviside step, while smooth bases need budget to catch up.A Haar wavelet is indistinguishable from the geometry-adaptive edge on the step because its piecewise-constant geometry matches the target.
  • Affine parameterisation: The exact tied-affine duality variant degrades more gently under noise but trails the independent-affine model on clean and small-sample data.The reported architecture therefore uses independent affine parameters despite the tied form's closer connection to exact duality.
Loading 2608.25807v1…