Source-linked AI summary
KAN 2.0: Kolmogorov-Arnold Networks Meet Science
Ziming Liu, Pingchuan Ma, Yixuan Wang, Wojciech Matusik, Max Tegmark
TL;DR
AI for curiosity-driven science must connect black-box learning with the symbolic and interpretable practices of science. KAN 2.0 builds a bidirectional framework and supporting tools to incorporate scientific knowledge into KANs and extract features, modular structures, formulas, and physical laws from them. The paper demonstrates this approach across conserved quantities, Lagrangians, symmetries, and constitutive laws.
Problem
Curiosity-driven science often lacks clear objectives, while existing KAN interpretability focuses narrowly on symbolic formulas and does not fully address embedding prior scientific knowledge.
Method
The paper develops bidirectional KAN–science methods, including MultKAN, kanpiler, and approaches for discovering important features, modular structures, and symbolic formulas.
Results
KANs are demonstrated on scientific discovery tasks involving conserved quantities, Lagrangians, hidden symmetries, and constitutive laws.
Takeaways & Limitations
The framework positions KANs as tools for both incorporating scientific knowledge and extracting interpretable scientific insights.
Takeaways & Limitations
KAN interpretability becomes increasingly difficult as model scale grows, so interpretability may remain practical mainly for relatively small networks.
Abstract
from arXiv · showhide
A major challenge of AI + Science lies in their inherent incompatibility: today's AI is primarily based on connectionism, while science depends on symbolism. To bridge the two worlds, we propose a framework to seamlessly synergize Kolmogorov-Arnold Networks (KANs) and science. The framework highlights KANs' usage for three aspects of scientific discovery: identifying relevant features, revealing modular structures, and discovering symbolic formulas. The synergy is bidirectional: science to KAN (incorporating scientific knowledge into KANs), and KAN to science (extracting scientific insights from KANs). We highlight major new functionalities in the pykan package: (1) MultKAN: KANs with multiplication nodes. (2) kanpiler: a KAN compiler that compiles symbolic formulas into KANs. (3) tree converter: convert KANs (or any neural networks) to tree graphs. Based on these tools, we demonstrate KANs' capability to discover various types of physical laws, including conserved quantities, Lagrangians, symmetries, and constitutive laws.
1 Introduction
The paper frames curiosity-driven science as requiring interpretable, interactive AI and proposes bidirectional synergy between KANs and scientific knowledge. It organizes scientific explanations around important features, modular structures, and symbolic formulas.
- Curiosity-driven science seeks deeper understanding rather than only predictions, creating different requirements for AI systems.
- AI tools for curiosity-driven research therefore need greater interpretability and interactivity to integrate into scientific practice.
- KANs support scientific interpretation through learnable edge functions and symbolic regression of decomposed one-dimensional functions.
- KAN 2.0 establishes bidirectional synergy by embedding scientific knowledge into KANs and extracting scientific knowledge from them.
- The framework treats scientific explanation as involving important features, modular structures, and symbolic formulas.
- The paper applies these ideas to conserved quantities, symmetries, Lagrangians, and constitutive laws through new KAN-based tools.
2 MultKAN: Augmenting KANs with multiplications
MultKAN extends KANs with explicit multiplication nodes, making multiplicative structures more direct and potentially more interpretable. On f(x, y) = xy, it learns a single multiplication node instead of the two-addition-node construction used by a KAN without multiplication.
- KAN background: KANs are built by stacking layers that transform vectors through learnable univariate functions and addition-based compositions.
- Motivation: A KAN without multiplication can represent products through additions combined with univariate functions, but the resulting computation may be difficult to recognize.
- MultKAN architecture: MultKAN inserts optional multiplication layers into KANs, with addition nodes copying subnodes and multiplication nodes multiplying selected subnode groups.
- MultKAN architecture: MultKAN widths separately record addition and multiplication operations, and the model reduces to a KAN when every multiplication width is zero.
- Implementation: Multiplication layers introduce no trainable parameters, so KAN sparse-regularization methods can be applied directly to MultKANs.
- Implementation: The implementation permits multiplication groups of any integer size k ≥2, although varying k values within a layer can challenge parallelization.
- Results: For f(x, y) = xy, MultKAN learns one multiplication node with linear learned activations, directly expressing the multiplication task.
3 Science to KANs
This section incorporates scientific knowledge into KANs through auxiliary features, modular structures, and symbolic formulas. It introduces tools for encoding these biases while retaining flexibility for data-driven discovery.
- Important features: Auxiliary variables can increase a KAN’s expressive power without adding information, reducing the resources needed to learn known domain structure.For relativistic mass, adding β and γ produced KANs with 2 or 3 edges and losses of 10^-6 and 10^-4, respectively.
- Important features: The auxiliary-variable relativistic-mass example shows that different random seeds can yield distinct solutions, including a sub-optimal approximation.The approximation 1 + (v/c)^2/2 = 1 + β^2/2 illustrates one such seed-dependent solution; hypothesis testing is proposed to compare alternatives.
- Modular structures: KAN modularity can encode separability and generalized symmetry, including multiplicative products over non-overlapping variable groups and hierarchical interactions through intermediate functions.The module method specifies node and subnode groupings so selected variable groups do not interact in designated layers.
- Symbolic formulas: The kanpiler converts symbolic formulas into KANs by parsing expression trees, aligning them with KAN architecture, and combining variables in the first layer.It was benchmarked on the Feynman dataset and successfully handled all 120 equations.
- Symbolic formulas: Compiled KANs exactly encode symbolic functions, while width and depth expansion can increase expressive power for subsequent data-driven fine-tuning.Expansion adds zero activation functions initially, so perturbation is required to make them trainable with non-zero gradients.
4 KANs to Science
KANs can extract scientific knowledge by identifying important features, revealing modular structures, and testing symbolic hypotheses. The paper develops attribution scores, pruning and modularity analyses, and checkpoint-based hypothesis testing for these goals.
- 4.1 Identifying important features from KANs: Attribution scores account for downstream network effects and better reflect variable or edge importance than local L1 norms.They are computed iteratively from outputs toward inputs, unlike scores based only on local activation norms.
- 4.1 Identifying important features from KANs: On two synthetic equations, attribution scores produce cleaner graphs and correctly identify equal importance where L1 norms misrank variables.For one equation, they suppress active first-layer edges that do not affect the final output; for another, they avoid incorrectly favoring x3 and x4.
- 4.1 Identifying important features from KANs: Attribution-based input pruning retains only relevant variables, turning a dense 100-input KAN into a more compact and interpretable network.In the example, only the first five features have significantly higher attribution scores, so prune_input retains those five.
- 4.2 Identifying modular structures from KANs: Anatomical modularity is induced by assigning neuron coordinates and permuting neurons to minimize connection cost while preserving network functionality.For multitask sparse parity, auto_swap identifies modules in both KANs and MLPs, while KANs discover simpler modules; for hierarchical majority voting, it succeeds for KANs.
- 4.2.2 Functional modularity: Functional modularity is analyzed through separability, generalized symmetry, and symmetry tests applied recursively to convert functions or networks into tree graphs.Both KANs and MLPs can reach correct tree graphs after sufficient training, but their intermediate training dynamics differ.
- 4.3 Identifying symbolic formulas from KANs: Checkpoint-based hypothesis testing compares alternative symbolic structures at equal complexity by pruning different edges and evaluating test loss.For the relativistic-mass example, removing β yields a 2.0×10−6 test RMSE, lower than the 6.5 × 10−4 obtained when removing γ.
5 Applications
The paper applies KANs to discover conserved quantities, Lagrangians, hidden symmetries, and constitutive laws. These applications combine differential-equation training, symbolic regression, prior knowledge, and model refinement to extract physical structure from data.
- Discovering conserved quantities: KANs recover the three conserved quantities of the 2D harmonic oscillator: x-direction energy, y-direction energy, and angular momentum.Three [4, [0, 2], 1] KANs trained with different random seeds correspond to H1, H2, and H3.
- Discovering Lagrangians: KANs learn interpretable Lagrangians by parameterizing L(q, q̇) and extracting symbolic functions from trained edge splines.The approach addresses LNN instability and limited MLP interpretability using KAN structure, symbolic initialization, and symbolic regression.
- Discovering Lagrangians: For a single pendulum, symbolic regression identifies the expected cosine potential and quadratic kinetic-energy terms.The cosine and quadratic functions rank among the five candidates best matching the learned splines.
- Discovering Lagrangians: For relativistic mass, symbolic regression finds V(q)=q but initially misses the compositional kinetic term, which is recovered after imposing a quadratic composition.The ground-truth x^-1/2 appears among the top candidates, while x^1/2 fits the imperfect spline slightly better.
- Discovering hidden symmetry: KANs and MLPs discover different Schwarzschild symmetry solutions: the MLP achieves 10^-8 loss with a smooth solution, whereas the KAN learns a domain wall.The KAN can be fine-tuned from the MLP solution to reach 10^-15 loss, close to machine precision.
- Learning constitutive laws: KAN symbolic refinement recovers constitutive laws, achieving a 6×10^-9 loss without prior knowledge but only a 7×10^-3 loss in the reported alternative case.The authors caution that prior knowledge can place optimization near a bad local minimum in this example.
6 Related works
Related work positions KANs within science-oriented neural networks, physical-law discovery, and mechanistic interpretability. KANs are presented as interpretable architectures because they decompose high-dimensional functions into one-dimensional functions.
- KANs and science-oriented networks: KANs extend the Kolmogorov-Arnold representation theorem to arbitrary-depth networks and have shown promise for accuracy and interpretability in science-oriented tasks.The architecture was proposed by Liu et al. and builds deeper networks by stacking KAN layers.
- Machine learning for physical laws: Previous machine-learning research has addressed equations of motion, conservation laws, symmetries, phase transitions, Lagrangians, Hamiltonians, and symbolic regression.The paper situates its contributions within a broad literature on learning physical laws from data.
- Mechanistic interpretability: KANs align with mechanistic-interpretability efforts because the Kolmogorov-Arnold theorem decomposes high-dimensional functions into collections of easier-to-interpret one-dimensional functions.Related work also studies inherently interpretable models and training methods that promote interpretability.
7 Discussion
The discussion frames KANs as a compromise between software-like interpretability and neural-network learnability. Efficiency improves substantially, but scalability limits both runtime advantages and interpretability, motivating larger-scale studies and stronger interpretability methods.
- Interpretability and learnability: KANs occupy an intermediate position between traditional software’s interpretability and neural networks’ learnability.The paper aims to make KANs more manipulable while retaining their ability to learn from data.
- Efficiency improvement: The pykan improvements reduce spline-evaluation memory from O(LN^2G) to O(LNG).Other changes skip unnecessary symbolic branches and intermediate activations.
- Efficiency improvement: GPU compatibility and implementation changes reduce one reported [4,100,100,100,1] training example from a full CPU day to 20 seconds on CPU and under one second on GPU.The paper still reports that KANs lag behind MLPs in efficiency, especially at large scales.
- Efficiency improvement: The package prioritizes interactivity and versatility over efficiency, including cached model data that consumes additional memory for convenient plotting.Users can call model.plot() without manually running a forward pass to collect data.
- Interpretability limitations: KAN interpretability becomes harder to manage as networks scale, although symbolic regression, modularity discovery, and feature attribution can extend the interpretability–scale frontier.The paper identifies more advanced interpretability methods as a future research direction.
- Future work: The framework is demonstrated primarily on small-scale, physics-related examples, with future work targeting larger problems and other scientific disciplines.This defines the current scope boundary of the paper’s demonstrations.