PaperlayerView exact revision on arXiv

Source-linked AI summary

Parameter symmetries determine representational geometry in overparameterized nonlinear networks

Marvin Theiss, Lukas Braun, Andrew M. Saxe, Erin Grant

arXiv:2609.39078v1cs.LG

TL;DR

Function can leave representations underdetermined because distinct parameterizations may compute the same function. The paper characterizes how parameter symmetries reshape representations and shows that suitable implementation-level constraints can select identifiable geometries.

  • Problem

    Distinct parameter configurations and neural-activity patterns can support the same function, leaving the extent to which function constrains representation in nonlinear networks unresolved.

    Source: §1 · p. 2

  • Method

    The paper characterizes function-preserving parameter symmetries through feature transformations and analyzes their effects on representational geometry and minimum-norm selection.

    Source: §1 · p. 2, §6 · p. 11

  • Results

    Symmetries act through addition, duplication, and scaling, while representational ambiguity can grow with overparameterization and minimum-weight-norm selection can identify a unique geometry.

    Source: §1 · p. 2, §7 · p. 12

  • Takeaways & Limitations

    Suitable implementation-level constraints can identify geometries in which features are weighted according to their contributions to the network’s function.

    Source: §1 · p. 2

    The main-text hidden-activation analysis focuses on positively 1-homogeneous activations, with other activation classes deferred to Appendix D.3.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Representations are routinely used across machine learning, psychology, and neuroscience to draw inferences about the computations of biological and artificial systems. Such inferences presume a meaningful link between representational geometry and the computation being performed. For artificial neural networks, however, the extent to which function constrains representation remains unclear. One key obstacle is that these networks admit parameter symmetries: changes in parameterization that preserve function exactly while reshaping representational geometry. Here, we show that a broad class of parameter symmetries acts on representations through just three primitive feature transformations: addition, duplication, and scaling. This feature-level characterization yields a closed-form decomposition of representational geometry into essential and auxiliary components, which makes precise how degeneracy in representational geometry can grow with overparameterization even when function is held fixed. Finally, we show that implementation-level selection rules can resolve this degeneracy, yielding identifiable geometries in which features are weighted according to their contributions to the network's function. Together, our results delineate when representations can support inferences about computation, and when they cannot.

1 Introduction

The introduction asks how much function constrains neural-network representations, given that distinct parameterizations can implement the same function. It characterizes symmetry-driven changes to representations and identifies implementation-level constraints that can restore identifiable geometries.

1 Introduction

  • Distinct parameter configurations and neural-activity patterns can support the same function, leaving open how strongly function constrains representation in nonlinear networks.The introduction frames this as a question about whether representational geometry can reliably inform analysis of computation.Source: §1 · p. 2
  • Function-preserving parameter symmetries reshape representations through compositions of three feature transformations: addition, duplication, and scaling.The paper refines symmetry classifications and derives their exact effects on representations.Source: §1 · p. 2
  • Implementation-level constraints can remove symmetry-induced degrees of freedom and select unique geometries whose features are calibrated by their contributions to the network’s function.The paper presents these conditions as sufficient for computation to constrain representation.Source: §1 · p. 2

2 Preliminaries and setting

The preliminaries set up fully connected one-hidden-layer networks and describe hidden activations as a matrix whose Gram matrix represents the geometry of responses to inputs.

2 Preliminaries and setting

  • The setting is a fully connected one-hidden-layer network with incoming weights, biases, readout weights, an activation function, and a parameterization.The parameterization is denoted by θ and includes the activation and network parameters.Source: §2 · p. 3
  • The hidden-activation matrix H collects the network’s hidden responses across inputs, with inputs as columns and neurons’ activity across inputs as rows.Each column is a population response to one input; each row records one neuron’s activity across inputs.Source: §2 · p. 3
  • The uncentered Gram matrix H^T H is the representational similarity matrix, encoding pairwise similarities between input-evoked population responses.It also decomposes into rank-one contributions from individual neurons, supporting analysis of how neuron contributions change under function-preserving transformations.Source: §2 · p. 3

3 Functional parameter symmetries in nonlinear networks

This section defines function-preserving parameter symmetries and the equivalence classes they generate, then characterizes symmetry types, irreducibility, and orbit invariants.

3 Functional parameter symmetries in nonlinear networks

  • Functional parameter symmetries preserve the network function over the entire input domain, and symmetry-equivalence is generated by finite compositions of these transformations.The paper focuses on this strict function-preserving notion when analyzing parameterizations and their representations.Source: §3 · pp. 3–4

3.1 A catalog of parameter symmetries

  • Symmetries include individual-neuron transformations, redundant zero and duplicate groups, and group transformations enabled by algebraic structure in the activation.Which forms are available depends on the activation; the classification includes even-linear and constant-odd forms as well as activations in neither class.Source: §3.1 · pp. 4–5

3.2 Symmetry orbits, irreducibility, and overparameterization

  • A symmetry orbit groups parameterizations connected by function-preserving symmetries; a parameterization is irreducible when no narrower member of its orbit exists.Irreducible members need not be unique, and can differ beyond generic permutation, positive-scaling, or sign-flip reparameterizations.Source: §3.2 · p. 5

3.3 Orbit invariants and essential parameter classes

  • For positively 1-homogeneous activations, symmetry-equivalence has a necessary-and-sufficient characterization based on invariant class contributions and residuals.The section defines essential classes as those with nonzero invariant coefficient.Source: §3.3 · pp. 5–6

4 Parameter symmetries act through three feature primitives

Across a symmetry orbit, parameter symmetries can be described through three feature operations, yielding a structural account of hidden activations.

4.1 Feature-level action of parameter symmetries

  • Parameter symmetries act on hidden activations through compositions of feature addition, duplication, and scaling, together with their inverses.These feature-level operations exhaustively characterize the effects of the symmetries on hidden activations.Source: §4.1 · p. 7
  • Zero-neuron and constant-neuron groups add features, duplicate-neuron groups duplicate them, and positive-scaling or sign-flip symmetries scale them.Linear-neuron and constant-neuron groups add oppositely oriented features, with duplication patterns determined by their aligned and opposite subgroups.Source: §4.1 · p. 7

4.2 Structure of hidden-activation matrices within a symmetry orbit

  • Within a positively 1-homogeneous activation's symmetry orbit, every hidden-activation matrix contains selected essential-class features, duplicated and scaled, alongside additional features.At least one orientation of every essential parameter class is represented; additional rows collect distinct features from nonessential classes and constant neurons.Source: §4.2 · p. 8
  • For several activation classes, every feature of an irreducible parameterization persists throughout its symmetry orbit up to nonzero scaling.These include activations that are neither even-linear nor constant-odd, and purely even or purely odd activations.Source: §4.2 · p. 8

5 Parameter symmetries dissociate function and representation

The section decomposes representational geometry into essential and auxiliary contributions, then characterizes how similarity to a fixed reference can vary within a symmetry orbit as width increases.

5.1 Essential and auxiliary contributions to representational geometry

  • The RSM separates into essential and auxiliary contributions: feature addition can add auxiliary rank-one terms, while duplication and scaling reweight terms.With positive-scaling symmetry, arbitrary strictly positive weights are possible while feature configuration, duplication pattern, and width remain fixed.Source: §5.1 · p. 9

5.2 Variability in representational similarity

  • The Pearson-correlation range against a fixed reference grows strictly with width until saturation; its limiting bounds depend on the activation, probe inputs, and reference geometry, not the computed function.Every orbit reaches the same limiting spread at finite width.Source: §5.2 · pp. 9–10
  • When rank(X) = P, every orbit can attain correlations arbitrarily close to 1 and −1 with any reference geometry; these limits hold at widths Nh ≥ N*.The rank condition is generic when P ≤ Ni + 1.Source: §5.2 · p. 10
  • If the reference RSM comes from a network with the same activation, the common upper similarity limit is 1 even when that network computes a different function.A feature can be added without changing the original computation, then amplified through duplication and scaling until its auxiliary contribution dominates the RSM.Source: §5.2 · p. 10

6 Identifiability through minimum-norm selection

Minimum-weight-norm selection removes auxiliary contributions and fixes the total weight assigned to each essential parameter class. Under additional conditions, the remaining orientation freedom vanishes and the resulting geometry is unique across widths.

6 Identifiability through minimum-norm selection

  • Minimum-weight-norm selection removes auxiliary contributions and fixes each essential class's total weight, leaving only residual-compatible allocation between its two orientations.The minimum objective value is 2 times the sum of the norms of the invariant class coefficients.Source: §6 · p. 11
  • If residual-compatible splits exist, m ≠ 0, and the class-specific residual matrices are linearly independent, all MWNPs have the same unique RSM at every larger width.The result assumes a positively 1-homogeneous activation.Source: §6 · pp. 11–12
  • The remaining orientation-split freedom disappears when the residual constraint uniquely determines the split.The paper gives linearly independent class-specific residual contributions as a condition under which this holds.Source: §6 · p. 11
  • All six analytical ReLU solutions in Figure 1 satisfy the identifiability conditions, yielding the same unique RSM within each orbit at every width Nh ≥ 2.This holds despite substantial representational variability within those orbits without the selection rule.Source: §6 · p. 12

7 Discussion

The discussion examines how function-preserving parameter symmetries create representational ambiguity under overparameterization, and how implementation-level efficiency constraints can restore a link between representation and function. It characterizes activation-independent and activation-dependent symmetries, including conditions for their function preservation and distinctness, while noting that these symmetries do not exhaust function-equivalent parameterizations.

7 Discussion

  • The paper characterizes how parameter symmetries reshape representational geometry without changing function, how ambiguity worsens with overparameterization, and how efficiency constraints can restore coupling.The authors describe the work as theoretical and connect its account of ambiguity to empirical dissociations reported in prior work.Source: §7 · p. 12
  • The results characterize representational variation generated by parameter symmetries but do not claim to exhaust the full set of parameterizations realizing a function.Earlier work studies identifiability within a layer; it does not address cross-layer degeneracies such as intermediate-layer representational collapse.Source: §7 · p. 12
  • Contravariance between task complexity and implementation dispersion cannot by itself explain representational universality when overparameterized models grow with task complexity.In this proportional regime, the implementation space need not narrow as tasks become more complex.Source: §7 · pp. 12–13
  • The framework distinguishes inter-individual representational differences that may be inconsequential from those reflecting computational differences, including noise robustness and transfer performance.The proposed normative selection rules can distinguish individuals along these computationally relevant dimensions.Source: §7 · p. 13

B Parameter symmetries in overparameterized nonlinear networks

  • The appendix establishes a necessary and sufficient criterion for determining whether network parameterizations are symmetry-equivalent.It treats generic reparameterizations and overparameterization-induced symmetries, including activation-independent and activation-dependent classes.Source: B Parameter symmetries in… · p. 25

B.1 Generic reparameterization symmetries

  • Permutation, positive scaling, and sign-flip symmetries preserve network function, with scaling requiring positive homogeneity and sign flips depending on activation symmetry.Permutation reorders hidden-layer neurons; sign flips apply when the activation is odd or even.Source: §B.1.1 · p. 25, §B.1.2 · p. 25, §B.1.3 · p. 26

B.2 Activation-independent overparameterization symmetries

  • Overparameterization-induced symmetries include zero-neuron groups, duplicate-neuron groups, and constant neurons, which rely on redundant neurons.In the cited teacher-student setting, these symmetries generate affine subspaces of equivalent networks; the appendix also imposes a nondegeneracy condition to exclude decomposable cases.Source: §B.2 · p. 26

B.2.1 Function-preserving realizations

  • Adding or removing a zero-neuron group, or replacing a neuron with a duplicate-neuron group, leaves the layer function unchanged.These are function-preserving realizations of the activation-independent overparameterization symmetries.Source: §B.2.1 · p. 27

B.3.6 Function-preserving realizations

  • Function-preserving realizations require residual affine or constant contributions to cancel; for even or odd activations, the corresponding groups are individually function-preserving.Merging duplicate-neuron copies preserves the function, while constant-neuron contributions must cancel for a constant-odd activation.Source: §B.2.1 · p. 27, §B.3.6 · pp. 31–32

B.2.2 Minimality and distinctness

  • The activation-independent taxonomy is minimal and mutually exclusive: zero-neuron, duplicate-neuron, and constant-neuron groups cannot be decomposed into smaller groups or confused with one another.Duplicate-group minimality is defined relative to a fixed reference readout.Source: §B.2.2 · pp. 27–28, §B.3.1 · p. 28

B.3 Activation-dependent overparameterization symmetries

  • Activation-independent symmetries arise from overparameterization alone, while activation-dependent symmetries require algebraic structure relating σ(z) and σ(−z).When activation-dependent symmetries exist, irreducible parameterizations can differ by more than generic reparameterization symmetries.Source: §B.3 · p. 28

B.3.1 Even-linear and constant-odd activations

  • Even-linear and constant-odd activations are defined by a linear odd component and a constant even component, respectively; affine activations are excluded.The two classes overlap only for affine activations, which are excluded so the classes are mutually exclusive in this setting.Source: §B.3.1 · p. 29

B.3.2 Aligned/opposite groups

  • Aligned/opposite groups comprise neurons whose incoming weights and biases agree up to sign flips; nondegeneracy conditions prevent these groups from splitting into smaller symmetries.Their nonempty aligned and opposite subgroups share reference parameters with opposite signs, and the conditions parallel subset-nonzero requirements for activation-independent groups.Source: §B.3.2 · p. 29

B.3.3 Symmetries arising from even-linear activations

  • Even-linear activations yield linear-neuron and linear-duplicate-neuron groups, while constant-odd activations yield constant-neuron and constant-duplicate-neuron groups.For even-linear activations, aggregate readouts control even and linear contributions; for constant-odd activations, they control constant and odd contributions.Source: §B.3.3 · p. 30, §B.3.4 · p. 30

B.3.5 Choice of sign in aligned/opposite groups

  • The aligned/opposite sign ambiguity is irrelevant for nonduplicate groups, remains formal but not algebraic for linear-duplicate groups, and is resolved by the definition for constant-duplicate groups.Duplicate groups retain a semantic orientation relative to the reference neuron.Source: §B.3.5 · p. 31

B.3.7 Minimality and nondegeneracy

  • Nondegeneracy conditions make the four activation-dependent symmetry classes minimal and mutually distinct from one another and from activation-independent classes.They exclude decompositions into smaller groups or degeneracies into other classes; duplicate variants additionally require a nonzero reference readout.Source: §B.3.7 · pp. 32–33

B.3.8 Nonuniqueness of irreducible parameterizations

This section examines nonuniqueness among irreducible parameterizations within a symmetry orbit, develops parameter-class invariants for symmetry-equivalence, and classifies common activations by even-linearity, constant-odd structure, and positive homogeneity. It notes that positively homogeneous activations are a subclass of even-linear activations, with only three `jax.nn` activations having degree-1 positive homogeneity.

B.3.8 Nonuniqueness of irreducible parameterizations

  • Symmetry-equivalent irreducible ReLU parameterizations can realize the same tent function yet remain unrelated by neuron permutations and positive scaling.The example also shows that activation-dependent symmetry groups may preserve the function only when added jointly, and its auxiliary-parameterization construction generalizes to a quantitative criterion for symmetry-equivalence.Source: §B.3.8 · p. 35

B.4.1 Parameter classes

  • Parameter classes group nonzero incoming parameters as all nonzero scalar multiples for degree-1 positively homogeneous activations, as ±w for even-linear or constant-odd activations, and otherwise as singletons.The class representation supports expressing the realized function in terms of parameter classes.Source: §B.4.1 · pp. 36–37

B.4.2 Characterization of symmetry-equivalence

  • Two parameterizations are symmetry-equivalent exactly when their aggregate coefficients match for every parameter class and their residual functions match.The proof constructs an auxiliary parameterization whose symmetry groups and residual contributions cancel, connecting parameterizations with these matching invariants.Source: §B.4.2 · pp. 38–40

C Symmetry properties of common activation functions

  • The appendix classifies every scalar activation in jax.nn v0.10.0 as even-linear, constant-odd, degree-1 positively homogeneous, or none of these, with categories allowed to overlap.The classification includes the identity activation for completeness, although affine activations are excluded from the paper’s nonlinear-network setting.Source: C Symmetry properties of common… · p. 41

C.1 Even-linear activations

  • The exact GELU and hard SiLU activations are even-linear but not constant-odd, while the identity activation is trivially even-linear.For identity, the even-linear slope is m = 1.Source: §C.1 · p. 41
  • Leaky ReLU and ReLU have linear odd components but are not constant-odd, whereas SiLU and sparseplus are even-linear.Source: §C.1 · pp. 43–45

C.2 Constant-odd activations

  • Softsign and tanh are odd, hence constant-odd with c=0 and not even-linear; hard sigmoid and sparse sigmoid are constant-odd with c=1, and sparse sigmoid is not even-linear.Source: §C.2 · pp. 45, 47

C.3 Positively homogeneous activations

  • For any positively homogeneous function of degree 1, on each half-line considered separately, the function is either identically zero or unbounded on that half-line.Positively 1-homogeneous activations form a subclass of even-linear activations.Source: §C.3 · p. 48

C.4 Activations with neither symmetry property

The appendices classify activations lacking the stated symmetry properties and formalize how parameter symmetries act on hidden features. They also characterize hidden-activation matrices within a symmetry orbit as duplicated and scaled essential features plus auxiliary rows.

C.4 Activations with neither symmetry property

  • ReLU6, SELU, CELU, ELU, and Mish are each neither even-linear nor constant-odd, based on their nonconstant even components and nonlinear odd components.Figure C.3 contrasts the identity with asymmetric activations and displays each activation's even and odd components.Source: §C.4 · pp. 49–51, Figure C.3 · p. 50

D.1 Duplication matrices

  • A duplication matrix repeats vector entries or matrix rows according to a positive-integer pattern, while its transpose repeats columns; weighted Gram diagonals sum weights within each duplication group.The duplication pattern's induced width is the resulting number of rows.Source: §D.1 · pp. 51–52

D.2 Feature-level characterization of parameter symmetries

  • Neuron groups implementing constant, linear, or duplicate-neuron symmetries translate into feature additions and duplications, while positive scaling and sign flips act as feature scalings.Reverse operations undo these primitives in reverse order; inverse duplication retains at least one copy of each original row.Source: §D.2 · pp. 54–55

D Feature transformations within symmetry orbits

  • Every parameter symmetry considered induces a composition of feature additions, duplications, scalings, or their inverses; some function-preserving operations must be combined.The decompositions are established for the cataloged symmetries, and jointly performed operations compose their feature-level transformations.Source: Table D.2 · p. 53, §D.2 · p. 55

D.3 Hidden activations within a symmetry orbit

  • Within an orbit, every hidden-activation matrix comprises duplicated and positively scaled represented essential features plus duplicated auxiliary features from nonessential classes or constant neurons.Every essential class is represented at least once; when the activation is not positively 1-homogeneous, the scaling factors may all be one.Source: §D.3 · pp. 56–57
  • Matching this factorization alone does not establish symmetry-equivalence; aggregate coefficients and residual functions must also coincide.Features in the auxiliary matrix need not be individually removable because they may contribute to the conserved residual.Source: §D.3 · p. 57

D.4 Persistence of irreducible features

Under specified activation assumptions, the section characterizes irreducible parameterizations within a symmetry orbit and shows that their feature rows persist across the orbit up to nonzero scaling. It also reports how representational ambiguity and similarity bounds vary within and across orbits.

D.4 Persistence of irreducible features

  • An irreducible parameterization has exactly one neuron per essential class, no nonessential-class neurons, and one constant neuron precisely when the global residual is nonzero.Under the stated assumptions, this structure characterizes irreducibility within the orbit.Source: §D.4 · p. 57
  • Irreducible parameterizations attain the orbit's minimum width, with one neuron per essential class and, when the residual is nonzero, one constant neuron.The width lower bound is attainable by a construction that reproduces the class aggregates and global residual.Source: §D.4 · p. 58
  • Every feature row of an irreducible parameterization appears in every parameterization in its orbit up to nonzero scaling.The persistence result is established by associating each essential-class feature, and any required constant feature, with neurons in the other parameterization.Source: §D.4 · pp. 60–61

E Representational geometry and similarity within symmetry orbits

  • Representational ambiguity increases strictly with width until reaching its supremum; limiting similarity bounds are orbit-independent and attained at finite width within every orbit.The section also studies conditions for approaching similarity to a reference geometry and gives an obstruction showing alignment is not always possible.Source: E Representational geometry and… · p. 61

E.1 RSM decomposition and positive reweighting

The RSM decomposition separates feature contributions and shows how scaling and feature addition can change geometry while preserving the realized function. With positively 1-homogeneous activations, increasing width expands attainable similarity ranges strictly until their orbit-wide bounds are reached.

E.1 RSM decomposition and positive reweighting

  • For the RSM decomposition, if activation σ is not positively 1-homogeneous, the factorization may be chosen with α = 1, in which case γ = ν.For activations covered by Corollary 4.3, the same calculation yields a decomposition in terms of features of any irreducible parameterization.Source: §E.1 · pp. 61–62
  • For positively 1-homogeneous activations, every strictly positive choice of effective weights is realizable with feature matrices, duplication pattern, and width fixed.The rescaling preserves each neuron's contribution to the realized function, so the new parameterization remains symmetry-equivalent.Source: §E.1 · p. 62

E.2 Strict growth until saturation

  • With positively 1-homogeneous activations, the attainable upper similarity bound rises strictly at each width until it reaches its orbit-wide limit.Attainable RSM sets are nested across increasing widths.Source: §E.2 · pp. 64, 67
  • The attainable lower bound falls strictly until its orbit-wide limit, so the similarity spread grows strictly until both bounds saturate.Growth occurs whenever at least one of the upper or lower bounds has not reached its orbit-wide limit.Source: §E.2 · p. 67

E.3 Function-independent similarity limits

The paper characterizes the limiting normalized geometries available within a symmetry orbit. Consequently, the limiting similarity bounds are common across orbits and are reached at finite width.

E.3 Function-independent similarity limits

  • Every symmetry orbit has the same limiting similarity bounds, which depend only on the activation, probe inputs, and reference geometry.The closure characterization identifies the normalized geometries underlying those bounds.Source: §E.3 · pp. 68, 70
  • Every orbit reaches the common limiting bounds and spread at a finite width.The proof characterizes the closure of attainable centered, Frobenius-normalized RSMs within each orbit.Source: §E.3 · pp. 69–70

E.4 Alignment with reference geometries

Alignment with a reference geometry depends on whether its centered, normalized form lies in the closed projected feature cone. Probe-input conditions can enable arbitrary-reference alignment, while activation and inputs can also obstruct it regardless of width.

E.4 Alignment with reference geometries

  • Similarity can approach one exactly when the centered, normalized reference geometry belongs to the closed projected feature cone.The criterion depends on the reference geometry's centered, normalized form and the cone determined by the activation and probe inputs.Source: §E.4 · p. 71
  • When the reference geometry comes from a network with the same activation, zero-readout feature addition followed by duplication and scaling can make its geometry dominate the RSM.This construction leaves the realized function unchanged.Source: §E.4 · pp. 71–72
  • If the probe inputs are affinely independent, the projected feature cone equals the full centered geometry space, enabling alignment with any reference geometry at the stated finite-width bound.The bound is Nh ≥ N⋆ + P(P−1)/2 − 1.Source: §E.4 · pp. 72–73
  • For ReLU networks on the specified one-dimensional probe inputs, the reference geometry from v = (1, 0, 1)ᵀ has nonpositive similarity at every width and for every realized function.The result holds whenever the correlation is defined.Source: §E.4 · p. 74

F Identifiability through minimum-norm selection

  • The following section analyzes minimum-norm selection, including how the finite input set affects representational identifiability and whether the minimum is attained.It treats minimum-weight-norm selection and then minimum-representation-norm selection.Source: F Identifiability through minimum-norm… · p. 75

F.1 Minimum weight norm for positively 1-homogeneous activations

For positively 1-homogeneous activations, minimum-weight-norm selection restricts geometries to those generated by feasible splits across essential feature orientations. Identifiability depends on the activation and residual constraints, while overparameterization can still leave multiple selected geometries.

F.1 Minimum weight norm for positively 1-homogeneous activations

  • Minimum-weight-norm RSMs are characterized by feasible splits across essential feature orientations, and every such split is realized by a minimizer.The feasible splits depend on width; their corresponding parameterizations attain the orbit-wide weight-norm lower bound.Source: §F.1 · pp. 76–77
  • Even when the minimum-weight-norm RSM is unique, duplication can redistribute coefficients without changing the RSM, so the parameterization need not be unique.Attainment of the lower bound requires the stated feasibility condition on the splits.Source: §F.1 · p. 77
  • When the activation has nonzero linear component and the residual contributions are linearly independent, the residual constraint fixes a unique split and therefore a unique minimum-weight-norm RSM at every larger width.All six analytical ReLU solutions considered have a unique such RSM for widths Nh ≥ 2.Source: §F.1 · pp. 77, 79
  • For purely even positively homogeneous activations with zero residual, all minimum-weight-norm parameterizations have the same RSM at widths Nh ≥ |E|.The two orientations generate identical features, so the geometry does not depend on the split.Source: §F.1 · p. 77
  • For the ReLU tent-function example, increasing width to Nh ≥ 6 permits a one-parameter family of distinct minimum-weight-norm geometries.The endpoint splits are feasible at width 3, while interior splits become feasible at width 6.Source: §F.1 · p. 78

F.2 Minimum representation norm for positively 1-homogeneous activations

For positively 1-homogeneous activations, minimum-representation-norm selection removes auxiliary contributions under the stated positivity and feasibility conditions, leaving residual-compatible orientation splits. Additional conditions make the resulting RSM unique and unchanged at larger widths.

F.2 Minimum representation norm for positively 1-homogeneous activations

  • At an attained minimum, each neuron's rank-one RSM contribution matches its sampled output contribution in magnitude, a neuron-wise balance that does not rule out cancellation.The balance condition does not imply uniqueness of the overall geometry.Source: §F.2 · p. 79
  • With g_q>0 and a feasible residual-compatible split, minimizers remove auxiliary contributions; essential weights depend on orbit invariants β_q and sampled feature norms g_q.The remaining freedom is allocation between orientations of equal feature norm.Source: §F.2 · pp. 80–82
  • All minimizers share one RSM when each class has a uniquely preferred orientation or the residual constraint uniquely determines the split; that RSM persists at every larger width.Identifiability holds from the first width at which a feasible split exists.Source: §F.2 · p. 82
  • If an essential orientation vanishes on sampled inputs (g_q=0), the representation-norm infimum can be unattained even though the function is preserved.The positivity condition rules out this nonattainment mechanism but is not necessary for a minimizer to exist in general.Source: §F.2 · p. 83

G Numerical and analytical characterization of ReLU networks solving the XOR task

  • The appendix identifies six qualitatively distinct ReLU solutions to XOR through a gradient-descent sweep, derives closed-form expressions in a geometric parameterization, and proves each can approach zero binary cross-entropy loss in the limit of parameter rescaling.The analysis uses a two-hidden-unit ReLU architecture and a binary cross-entropy training objective.Source: G Numerical and analytical… · pp. 83–84

G.1 Setup

  • The XOR setup uses same-sign inputs labeled 0 and opposite-sign inputs labeled 1; each nonzero hidden neuron is parameterized by boundary angle, signed distance, and gain.The activation boundary is a line with unit normal n(φ) at signed distance d from the origin.Source: §G.1 · p. 84

G.2 Gradient-descent sweep and solution clustering

A gradient-descent sweep found six dominant two-unit ReLU solution types for XOR, with closed-form representatives in two boundary-orientation families. Each classifies the four points correctly, and scaling its parameters sends BCE loss toward zero without attaining zero at any finite parameter vector.

G.2 Gradient-descent sweep and solution clustering

  • 296 of 1000 runs converged, clustering into six solution types with counts 88, 83, 34, 34, 32, and 25; 19 distances and two angles were atypical.The six types formed diagonal and anti-diagonal families; the sweep does not claim to exhaust solutions under other training protocols.Source: §G.2 · pp. 84, 86, 87, Figure G.1 · p. 85

G.3 Closed-form characterization of the six dominant solution types

  • All six closed-form solutions fall into diagonal and anti-diagonal families, reduce inputs to one projection, and assign every XOR point a logit magnitude of 1/√2.The six solutions share the same logits on the four data points despite differing boundary positions and readout weights.Source: §G.3 · pp. 87–88, Figure G.3 · p. 88

G.4 Approaching zero BCE loss by parameter scaling

  • No finite parameter vector attains zero BCE, but scaling each solution's incoming parameters and output bias makes its loss decrease monotonically toward zero.Along the scaling ray, the logits are multiplied by the scale factor.Source: §G.4 · p. 89

Related papers

Loading 2609.39078v1…