PaperlayerView exact revision on arXiv

Source-linked AI summary

A Class of R\'enyi-based Concentration Metrics for Kolmogorov-Arnold Geometry in Multilayer Perceptrons

Juan Carlos Fernández, Asunción Jiménez-Cordero, José Francisco Paris

arXiv:2610.23267v1cs.ITmath.ITstat.ML

TL;DR

The Participation Ratio’s maximum varies with network architecture, making its values difficult to interpret consistently across architectures. The paper proposes Rényi-based α-RC metrics that compare normalized minor magnitudes with a uniform distribution. In single-hidden-layer MLP experiments, training increases concentration across widths, orders, and α values, while its relationship with predictive performance is not monotone.

  • Problem

    The Participation Ratio’s maximum depends on network architecture, so its values do not have a consistent concentration interpretation across architectures.

    Source: §1 · p. 3

  • Method

    The α-RC metrics compare the normalized distribution of minor magnitudes with the uniform distribution using Rényi divergence, scaled to have endpoints 0 and 1.

    Source: §1 · p. 4, §5 · p. 36

  • Results

    Training increases concentration for every tested width, order k, and α, with larger increases for larger k, while concentration and predictive performance do not change monotonically together.

    Source: §5 · p. 37

  • Takeaways & Limitations

    The α-RC metrics give concentration values whose endpoints retain the same meaning across architectures, and at α = 2 the metric is a transformation of the Participation Ratio.

    Source: §5 · pp. 36–37

    Exact computation is infeasible for realistic networks because the number of minors depends on network width and order k.

    Source: §5 · p. 37

Abstract

from arXiv · show

The interpretability of artificial intelligence models has become a central concern, especially for black-box models such as multilayer perceptrons (MLPs). Kolmogorov-Arnold Geometry has recently emerged as a framework for analysing the internal representations of these networks, by examining how the hidden layer locally responds to changes in the input through the minors of its Jacobian. The degree to which a few of these minors dominate the rest, known as their concentration, has so far been measured with the Participation Ratio. Although this metric has useful properties, its maximum value depends on the size of the network, so a given value cannot be read as high or low concentration without reference to the architecture. To overcome this limitation, in this paper we propose the $\alpha$-RC metrics, a family of R\'enyi-based concentration measures that compare the normalized magnitudes of the minors with the uniform distribution through the R\'enyi divergence of order $\alpha$. All the metrics in the family are bounded between 0 and 1 regardless of the architecture, with 0 indicating that the magnitude is evenly spread over the minors and 1 that it is carried by a single one. Moreover, the metrics are non-decreasing in $\alpha$, and for $\alpha$ = 2 they reduce to an explicit transformation of the Participation Ratio. Computational experiments with single-hidden-layer MLPs trained on a synthetic dataset show the benefits of our proposal.

1. Introduction

Kolmogorov–Arnold Geometry examines hidden representations through Jacobian minors, whose concentration has been measured with the Participation Ratio (PR). Because the PR’s attainable maximum varies with network architecture, the paper introduces Rényi-based metrics with architecture-independent endpoints and illustrates them on trained MLPs.

1. Introduction

  • Kolmogorov–Arnold Geometry analyzes hidden representations through Jacobian minors, whose concentration indicates whether a few minors carry most of the total magnitude.Prior experiments associated increased concentration with better performance in some training regimes, but not others.Source: §1 · p. 3
  • The PR’s maximum grows with the number of hidden neurons, so the same value can indicate different concentration levels across architectures.A ratio of trained-model PR to initialization PR tracks change during training but does not give the PR fixed endpoint meanings.Source: §1 · pp. 3–4
  • The proposed α-RC metrics compare normalized minor magnitudes with the uniform distribution using Rényi entropy and have architecture-independent bounds of 0 and 1.Zero denotes uniform magnitudes; one denotes all magnitude carried by a single minor.Source: §1 · p. 4
  • The α-RC metrics are non-decreasing in α, which shifts emphasis from the proportion of non-zero entries at small orders toward the distribution’s mode at large orders.At α = 2, the metric is a function of the Participation Ratio.Source: §1 · p. 4
  • Experiments on single-hidden-layer MLPs trained on a continuous XOR problem illustrate the architecture-independent scale and examine how α affects the metric values.The paper contrasts this scale with the PR’s architecture-dependent interpretation.Source: §1 · p. 4

2. The Kolmogorov–Arnold Geometry

Kolmogorov–Arnold Geometry characterizes local changes in hidden representations using Jacobians and their minors. The section connects that geometry to MLPs and reviews the Participation Ratio as a measure of minor concentration.

2. The Kolmogorov–Arnold Geometry

  • The section frames KAG as a way to characterize how the inner map’s local geometry reflects hidden representations.Its subsections develop the geometric intuition, Jacobian and exterior-power tools, MLP specialization, and PR measure.Source: §2 · pp. 4–5

2.1. From the Kolmogorov–Arnold theorem to Kolmogorov–Arnold Geometry

  • The Kolmogorov–Arnold representation theorem expresses continuous multivariate functions as a composition of an inner map and an outer map, and KAG studies the inner map’s local geometry.The inner map is universal in the theorem’s construction, while the outer map depends on the target function.Source: §2.1 · pp. 5–6

2.2. Jacobians, exterior powers and minor concentration

  • The Jacobian describes local hidden-coordinate responses, while its k-th exterior power collects k × k minors that measure joint changes in k-dimensional volume.A minor’s absolute value measures the volume-change magnitude, and its sign indicates orientation.Source: §2.2 · pp. 6–8

2.3. MLP and its Kolmogorov–Arnold Geometry

  • In the studied single-hidden-layer MLP, an input-dependent near-zero activation derivative makes every minor involving that hidden neuron approximately zero.The network uses a smooth GeLU activation; its Jacobian is defined everywhere, unlike the discussed Kolmogorov–Arnold map, which is guaranteed differentiable only almost everywhere.Source: §2.3 · pp. 9–11

2.4. Participation Ratio

  • The Participation Ratio measures whether minor magnitudes are concentrated in a few entries or spread across many, but its maximum depends on the number of minors.It is undefined when the order-k exterior power is zero, including when the Jacobian rank is below k.Source: §2.4 · pp. 11–13

3. A novel improved metric

The section presents Rényi-based concentration scores that compare normalized minor magnitudes with uniformity, addressing Participation Ratio’s architecture-dependent scale with architecture-independent endpoints and a tunable order α. At α = 2, the metric is linked to a normalized PR that preserves its geometric interpretation and can distinguish networks that PR rates identically.

3.1. Motivating example

  • The same PR value can signify uniform activity in a smaller network but concentrated activity in a larger one, making comparisons across widths misleading.The motivating example contrasts uniform activity across 2 minors with activity concentrated in 2 of 16 minors.Source: §3.1 · pp. 14–15

3.2.1. Normalization of Exterior Power Values

  • The proposed score treats absolute minor magnitudes normalized by their ℓ1 norm as a discrete PMF and compares that distribution with uniformity, with values from 0 to 1.Zero denotes uniform magnitudes and one denotes a single nonzero minor, regardless of architecture.Source: §3.2 · p. 16, §3.2.1 · pp. 16–17

3.2.2. Rényi Divergence and Its Properties

  • Rényi divergence is nonnegative and, for α > 0, vanishes only when the compared distributions coincide; against uniformity, it can be expressed through Rényi entropy.The paper uses divergence from the uniform distribution to quantify concentration.Source: §3.2.2 · p. 20

3.2.3. Formal Definition of the Class of α-RC Metrics

  • The α-RC metrics lie in [0, 1] and are non-decreasing as α increases, with equality between orders exactly when the minor distribution is uniform on its support.The monotonicity result assumes 1 ≤ k ≤ min(m, n) and at least two order-k minors.Source: §3.2.3 · p. 22

3.2.4. Connection of the 2-RC Metric with the Participation Ratio

  • The 2-RC metric has an exact closed-form connection to the Participation Ratio.The paper identifies 2-RC with the collision-entropy case.Source: §3.2.3 · p. 22, §3.2.4 · p. 23
  • The 2-RC metric normalizes PR to remove its architecture-size-dependent range while preserving its intuitive geometric interpretation.Source: §3.2.4 · p. 23

3.2.5. Illustrative Example with the 2-RC Metric

  • PR gives identical values for a fully even 2-minor network and a 16-minor network with two nonzero entries, but 2-RC captures concentration in 2/16 of the latter’s capacity.Source: §3.2.5 · p. 24

4. Computational experiments

Computational experiments on single-hidden-layer MLPs trained on continuous XOR examine predictive performance and minor concentration across widths, orders, and training using α-RC metrics on a common 0-to-1 scale. Concentration increased during training and more for higher orders, but remained intermediate and did not track predictive performance monotonically.

4.2. Predictive performance

  • Test R2 ranges were 0.3869–0.7592 for m = 8 and 0.8519–0.9660 for m = 32; training and test results differed by less than 0.017 for both widths.The narrow network was more sensitive to the seed, and the near-identical training and test results showed no sign of overfitting.Source: §4.2 · pp. 28–29, Table 2 · p. 28

4.5. Discussion

  • C(k)_α increased monotonically with α at the fixed input and on average, while the order-2 metric gives concentration on a 0-to-1 scale unlike the architecture-dependent Participation Ratio bound.For m = 32 and k = 3, mean C(3)_2 rose from 0.1216, near uniformity, to 0.6384, an intermediate concentration level.Source: §4.3 · p. 30, §4.5 · p. 35

4.3. Pointwise concentration at a fixed input

  • For m = 32, mean C(3)_2 rose from 0.1216 to 0.6384 across training for k = 3, and concentration increased for every order and α while Participation Ratio decreased.At the fixed test point, concentration was lower than the test-set average in every table row, showing that this point alone did not represent the test set.Source: §4.3 · pp. 30–31, Table 3 · p. 30

4.4. Evolution of the concentration during training

  • For m = 32 and k = 3, concentration rose rapidly in early training, peaked around epochs 210–240, then declined slightly as test R2 continued to increase.The peak roughly coincided with the maximum number of hidden neurons whose input weights were below 10^-8 times the largest entry of A; their minors became negligible relative to those of the remaining neurons.Source: Figure 1 · p. 32, §4.4 · pp. 32, 34, Figure 2 · p. 34

4.5. Discussion

  • For m=32, concentration growth accompanies and slightly precedes learning, then declines slightly while test R2 improves; its link to performance depends on training regime (Freedman & Mulligan, 2025).Source: §4.5 · p. 35
  • At epoch 500 for m=32, k=3, mean ∞-RC metric C(3) is 0.7556, while mean C(3) at α=1/2 is 0.4198 and never exceeds 0.4763.The magnitude concentrates on a subset of minors without collapsing onto one, and no minor vanishes.Source: §4.5 · pp. 35–36
  • Higher orders show larger concentration increases because each minor multiplies derivatives of k hidden neurons, amplifying suppression when some derivatives become small.Source: §4.5 · p. 35
  • Across four evaluated orders, concentration increases with training and more for higher orders; α selects distribution emphasis, with α=1/2 weighting small minors and α=∞ depending only on the largest.By monotonicity, α=1/2 and α=∞ bound intermediate orders pointwise; α=0 depends only on vanishing minors and is constant for the GeLU MLP.Source: §4.5 · p. 36

5. Conclusions and Future Research

The paper introduces α-RC metrics that put minor concentration on an architecture-independent 0–1 scale, and uses single-hidden-layer MLP experiments to illustrate their behavior. Exact computation is not feasible for realistic networks, so efficient estimation remains needed.

5. Conclusions and Future Research

  • The α-RC family uses Rényi divergence from the uniform minor distribution, is bounded by 0 and 1 across architectures, and is non-decreasing in α.For α = 2, it is a closed-form transformation of the Participation Ratio; α selects which part of the distribution is summarized.Source: §5 · pp. 36–37
  • Training increases concentration across widths, orders k, and α values, with larger increases as k grows.For m = 32, the α = 2 metric rises from 0.12 at initialization to a maximum of 0.66; concentration begins increasing before the coefficient of determination departs from zero, and later decreases slightly while performance improves.Source: §5 · p. 37
  • C(∞) remains below 0.78 throughout training, indicating concentration on a subset of minors rather than a single minor.The conclusion describes the observed concentration as intermediate.Source: §5 · p. 37
  • The proposed metrics have been applied to networks in which all minors can be enumerated.The paper suggests they could also monitor or guide training, for example through a concentration term in the loss.Source: §5 · p. 37
  • Exact computation is not feasible for realistic networks, making efficient estimation methods necessary.The paper also notes that the number of minors grows combinatorially with network width and input dimension.Source: §5 · p. 37

CRediT authorship contribution statement

The CRediT statement assigns conceptualization and methodology to all three authors, with additional roles distributed across supervision, software, validation, and writing.

CRediT authorship contribution statement

  • Juan Carlos Fernández contributed conceptualization, methodology, software, and writing; Asunción Jiménez-Cordero contributed conceptualization, methodology, supervision, validation, and writing; José Francisco Paris contributed conceptualization, methodology, supervision, validation, and review and editing.Fernández and Jiménez-Cordero are credited with the original draft; all three are credited with review and editing.Source: CRediT authorship contribution statement · p. 38

Data availability

The authors make the computational materials available on GitHub.

Data availability

  • The implementation code, synthetic datasets, trained-network weights at every epoch, and computed C(k) values are available on GitHub.The repository is identified as JuanCarlosFdezMlg/renyi-concentration-kag.Source: Data availability · p. 38

Declaration of generative AI and AI assisted technologies in the manuscript

The authors report using Claude and Codex to assist with implementation and manuscript preparation, then reviewing and editing the resulting content.

Declaration of generative AI and AI assisted technologies in the manuscript

  • Claude and Codex assisted with experiment-code implementation, drafting and revising text, and improving manuscript language and readability; the authors reviewed and edited the content and take responsibility for it.Source: preparation process · pp. 38–39
Loading 2610.23267v1…