PaperlayerView exact revision on arXiv

Source-linked AI summary

Per-Node Activation Function Evolution in Indirectly Encoded Substrates: Solvability, Limits, and Emergent Diversity

Romain Claret, Michael O'Neill, Paul Cotofrei, Kilian Stoffel

arXiv:2610.00149v1cs.NEcs.AI

TL;DR

The paper asks which CPPN-based assignment mechanism best selects per-node functions and whether the solvability divide is specific to particular problem structures and optimizers. The authors evolve CPPNs that generate substrate connection weights and, through an additional output, assign each node an activation function from a palette. On Parity-4, four oscillatory functions achieved 100%, five functions formed an intermediate tier, and all 9 remaining strictly monotonic functions achieved 0%; the divide varies by task, architecture, and optimizer.

  • Problem

    The paper asks which CPPN-based assignment mechanism best selects per-node functions and whether the solvability divide is specific to particular problem structures and optimizers.

    Source: Introduction · p. 1

  • Method

    The authors evolve CPPNs that generate substrate connection weights and, through an additional output, assign each node an activation function from a palette.

    Source: System Pipeline · p. 2, Experimental Setup · p. 3

  • Results

    On Parity-4, four oscillatory functions achieved 100%, five functions formed an intermediate tier, and all 9 remaining strictly monotonic functions achieved 0%; the divide varies by task, architecture, and optimizer.

    Source: Per-Function Ablation · p. 4, Conclusion · p. 8

  • Takeaways & Limitations

    Per-node activation selection in indirect encoding lets evolution search a design space impractical to choose by hand, shaping substrates through function category, problem geometry, and optimizer.

    Source: Conclusion · p. 8

    The feedforward results concern evolved substrates with few hidden nodes; although monotonic functions can represent parity with sufficient width, neuroevolution did not discover that construction in these experiments.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Biological neurons achieve computational diversity through specialized types: tonic, bursting, adapting, and fast-spiking cells coexist within the same circuit. Artificial neural networks, by contrast, apply a single activation function uniformly to all nodes, which limits what they can represent. We show that this uniformity creates hard limits for evolutionary search: across sparse evolved substrates, monotonic functions fail to solve parity beyond its smallest instance, XOR, while a single oscillatory unit suffices at all tested scales. The gap is one of search and sparsity, not representation: monotonic networks can represent parity with a modest number of hidden units, and gradient descent recovers that solution. We evolve, to our knowledge for the first time in indirect encoding, per-node activation function assignments from an 18-function palette across more than 4,500 experimental runs spanning Boolean logic, regression, and spatial classification. Testing each of the 18 functions individually on Parity-4 reveals a three-tier solvability structure: oscillatory functions achieve 100%, intermediate functions 6.7-80%, and all 9 monotonic functions 0%. This divide is not universal. Recurrence collapses it, and gradient descent inverts it entirely, showing that the barrier is specific to evolutionary search in sparse substrates. What activation functions a network can use, beyond its topology and weights, determines what evolutionary search can solve. Indirect encoding discovers heterogeneous per-node activation assignments unlikely to be chosen by hand.

Introduction

The introduction frames two questions: which mechanism best selects per-node functions via CPPN, and whether the solvability divide depends on problem structure or optimizer.

  • Which assignment mechanism is most effective for per-node function selection via CPPN?Source: Introduction · p. 1
  • Is the solvability divide universal, or specific to particular problem structures and optimizers?Source: Introduction · p. 1

1. Per-function solvability

The section characterizes function-wise solvability and tests whether the divide extends beyond parity or changes under different conditions.

Per-function solvability

  • On Parity-4, the palette revealed a three-tier solvability structure spanning 100%, 6.7–80%, and 0%, with local extrema count quantitatively predicting the tiers (Spearman ρ=0.81).Source: §1 · p. 2

Assignment method and palette robustness

  • CPPN-controlled per-node assignment significantly outperformed heuristic and uniform methods; larger palettes slowed convergence without reducing success.The divide persisted across population sizes 50–2,000.Source: §1 · p. 2

Boundary conditions

  • The divide was not universal: recurrence collapsed it, gradient descent inverted it, and all 18 functions solved sine regression at 100%.Source: §1 · p. 2

Background

The background traces indirect encoding from CPPN-generated substrates to a method for assigning heterogeneous activation functions to substrate nodes.

Indirect Encoding and HyperNEAT

  • HyperNEAT uses a CPPN to generate connection weights from spatial coordinates, while ES-HyperNEAT adds variance-based neuron placement.EMR-HyperNEAT replaces sequential quadtree processing with an eagerly evaluated dense multi-resolution grid for GPU acceleration.Source: Indirect Encoding and HyperNEAT · p. 2
  • Standard HyperNEAT applies tanh uniformly to substrate nodes; this work extends it with per-node selection from 18 activation functions.Source: Indirect Encoding and HyperNEAT · p. 2

Activation Function Diversity

  • Prior activation-diversity approaches evolve per-node functions via direct encoding, whereas this method assigns them indirectly from spatial coordinates.The CPPN output creates spatially structured functional regions without per-gene mutation.Source: Activation Function Diversity · p. 2
  • The authors report no prior work evolving per-node substrate activations through indirect encoding and provide formal analyses of both sides of the divide.Source: Activation Function Diversity · p. 2

Related Work

Related work situates the study among heterogeneous-activation, periodic-activation, and architecture-search approaches.

  • Earlier mixed-activation work linked diversity to parsimony, while this study reports a categorical solvability divide beyond a size advantage.Prior per-node activation work used direct encoding.Source: Related Work · p. 2
  • Periodic-activation studies connect oscillatory units with implicit representations, XOR decision boundaries, and arbitrary Boolean computation.The cited Growing Cosine Unit work reports a single oscillatory neuron solving XOR (Noel et al., 2021).Source: Related Work · p. 2
  • Architecture searches may include activations at the layer level, whereas this work assigns functions per node through indirect encoding.Source: Related Work · p. 2

Methods

The method evolves a CPPN that generates substrate weights and per-node activations, then evaluates the substrate under NEAT selection. It compares three assignment schemes from an 18-function palette across several task families and replicated settings.

System Pipeline

  • A CPPN generates substrate weights and assigns each node an activation before the substrate is evaluated and NEAT selection produces the next generation.Source: System Pipeline · p. 2

Three Per-Node Assignment Methods

  • CPPN-output bins an extra CPPN channel into palette indices, while weight-interpretation maps incoming-weight statistics to indices and uniform assignment gives all nodes one function.Weight-interpretation includes sign-, magnitude-, and variance-based variants.Source: Three Per-Node Assignment Methods · p. 3

Function Palette: 18 Activation Functions

Experimental Setup

  • Experiments cover logic, classification, regression, and Tower of Hanoi, with 30 replications and stated solve thresholds; defaults are population 150 and 300 generations.Source: Experimental Setup · pp. 3–4
  • Each individual is a NEAT-evolved CPPN that emits weights and, in CPPN-output mode, an activation-index channel; variance filtering retains informative grid nodes.Group comparisons use Mann–Whitney U or Fisher’s exact tests, and correlations use Spearman’s rank coefficient.Source: Experimental Setup · pp. 3–4

Results

On XOR, sinusoidal activation solves faster than a monotonic-only palette, while sparse evolved substrates do not discover the width-Θ(n) monotonic parity construction. Formal results distinguish limits of evolutionary search in sparse substrates from representational limits: one cosine unit suffices, while monotonic networks can represent parity with width Θ(n).

Per-Function Solvability

  • 1.1 vs 17.5 generations: sin solves XOR 16× faster than the monotonic-only baseline, and the contrast with Gaussian supports periodicity—not smoothness—as the decisive property.Sin converges in 1.1 generations versus 17.5 for the tanh, sigmoid, relu, and identity baseline; sin achieves 100% on parity while Gaussian achieves 33%.Source: Per-Function Solvability · p. 4
  • In sparse evolved substrates, variance-based discovery yields too few hidden nodes for the width-Θ(n) monotonic construction, which evolution does not discover.Source: Per-Function Solvability · p. 4

Formal Analysis

  • A single hidden cosine unit computes n-bit parity for any n by encoding the input count as an alternating sign and applying a linear readout.The readout maps the cosine output to parity.Source: Formal Analysis · p. 4
  • A single monotonic unit cannot compute parity for n ≥2, but a single hidden layer of monotonic units can do so with width Θ(n).The cited proposition attributes the single-unit result to Minsky and Papert (2017).Source: Formal Analysis · p. 4
  • The divide concerns discovery in sparse evolutionary search: variance-based substrates yield few hidden nodes, while gradient descent with a fixed wide layer reliably reaches the monotonic solution.Source: Formal Analysis · p. 4

Per-Function Ablation

Per-function tests identify a three-tier Parity-4 solvability pattern linked to oscillatory structure, while parity scaling and larger populations preserve the sin–tanh gap. More search budget improves tanh’s best fitness but does not make it solve larger parity instances or Parity-4.

Per-Function Ablation

  • On Parity-4 (N=30), local extrema count predicts solve rate (Spearman ρ=0.81, p=4.4×10−5): four functions achieve 100%, five form an intermediate tier, and all nine strictly monotonic functions achieve 0%.The two 2-extrema outliers differ in extrema amplitude: band pass has deep sign-reversing extrema and solves at 100%, whereas integrate has shallow perturbations and solves at 0%.Source: Per-Function Ablation · p. 4, Figure 2 · p. 5

Parity Scaling

  • Across Parity-2 through Parity-8, sin solves 100% of runs, while tanh solves 50% on Parity-2 and 0% from Parity-3 onward.Sin averages 1–4 generations to solve; tanh’s mean best fitness falls from 0.930 on Parity-2 to 0.791 on Parity-8.Source: Parity Scaling · p. 5, Table 2 · p. 5

Population Size Invariance

  • Across population sizes 50–2,000, Parity-4 solve rates remain 100% for sin and 0% for tanh, although tanh’s best-found fitness rises from 0.808 to 0.910.Sin’s median convergence falls from 5 generations at population 50 to 1 at populations ≥1,000; tanh never reaches the 0.95 threshold.Source: Population Size Invariance · p. 5, Table 3 · p. 5

Assignment Method Comparison

CPPN-controlled function assignment shows task-dependent effects, and larger palettes preserve Parity-4 success while slowing convergence. Recurrence partly changes the parity divide, with effects that vary by activation and problem.

Assignment Method Comparison

  • On circles, CPPN-output reaches 66.7% vs 0% for both weight-interpretation and uniform; on Two Spirals, adding sin lowers performance (76.1% vs 76.9%).Weight-interpretation matches the 75.2% uniform baseline on Two Spirals, while CPPN-output provides only a marginal gain there.Source: Assignment Method Comparison · pp. 5–6, Table 4 · p. 6

Palette Size Robustness

  • Parity-4 success stays 100% for palette sizes 1–13, while convergence slows 5.5× (1.7→9.3 generations) and standard deviation rises from 0.7 to 6.4.The tested palettes draw from the 18-function activation set.Source: Palette Size Robustness · p. 6, Table 5 · p. 6

Recurrence Collapses the Divide

  • On Parity-4, recurrence raises some bounded monotonic solve rates, including softplus from 0% to 90% and sigmoid from 0% to 70%, while tanh reaches only 6.7%.Oscillatory functions remain at 100%; identity and lelu remain at 0%.Source: Recurrence Collapses the Divide · p. 6, Table 6 · p. 6
  • On Parity-8 with recurrence, bounded or radial functions partially solve at 32.2%, while sigmoid reaches 70%, softplus 50%, and gauss 27%; unbounded or linear functions remain at 0%.Source: Recurrence Collapses the Divide · p. 6
  • For recurrent tanh on Parity-4, hidden-to-hidden connections alone solve 20% of runs versus 0% feedforward; backward, lateral, and self-loops add little (6.7–13.3%).Source: Recurrence Collapses the Divide · p. 6, Table 7 · p. 6
  • Recurrence solves 0% of Tower of Hanoi and Two Spirals runs, but degrades most functions on circles; oscillatory adapt alone stays at 100%.On circles, pure sin reaches 80% and resonator 70%; the authors relate the contrast to temporal iteration helping parity but disrupting spatial coherence.Source: Recurrence Collapses the Divide · p. 7, Table 8 · p. 7

Non-Parity Generalization

Across non-parity tasks, activation-function performance depends on the decision boundary: all functions solve sine regression, while circles and the step function favor different activation types.

Non-Parity Generalization

  • All 18 activation functions achieve 100% on sine regression, where smooth interpolation replaces discrete boundary partitioning.Source: Non-Parity Generalization · p. 7, Table 9 · p. 7
  • On concentric circles, oscillatory functions solve at 100%, while 7 of 9 monotonic functions reach 0%; sigmoid and relu each reach 3.3%.Source: Non-Parity Generalization · p. 7, Table 9 · p. 7
  • On the step function, identity converges fastest at a median of 4 generations, monotonic functions outperform oscillatory ones, and gauss solves 13%.Sin takes a median of 55 generations on this linear classification task.Source: Non-Parity Generalization · p. 7, Table 9 · p. 7
  • The divide appears on problems requiring non-monotonic decision surfaces and disappears on smooth problems; oscillatory structure is counterproductive for a linearly separable boundary.Source: Non-Parity Generalization · p. 7

Discussion

With gradient descent, the evolutionary oscillatory/monotonic divide reverses. The authors recommend matching activation functions and assignment mechanisms to task structure, while bounding conclusions to the tested settings.

The Divide Is Optimizer-Specific

  • From Parity-6 onward, tanh stays at 100% through Parity-8 while sin falls to 0%, reversing the evolutionary divide.On Parity-6, increasing hidden width from h=16 to h=32 lowers sin performance from 66.7% to 46.7%; the authors attribute this to near-zero pre-activations under Xavier initialization.Source: The Divide Is Optimizer-Specific · p. 7, Table 10 · p. 8

Practical Implications

  • For multi-crossing tasks, include oscillatory functions; on sine regression, all 18 functions reach 100%, and osc adapt stays at 100% across tested conditions.The authors recommend osc adapt as a safe default across tested parity, circles, feedforward, and recurrent conditions.Source: Practical Implications · pp. 7–8
  • Use a dedicated CPPN output for function selection rather than deriving assignments from weights.The authors report CPPN-output assignment outperforms weight-interpretation and uniform methods.Source: Practical Implications · p. 8
  • On Parity-4, expanding the palette from 1 to 13 functions slows convergence 5.5× without reducing success; use the full palette when task structure is unknown.The authors frame the slower convergence as insurance against choosing the wrong category.Source: Practical Implications · p. 8

Limitations

  • Evidence covers problems with at most 8 inputs; high-dimensional tasks such as vision or robotics may blur the divide.Prior ES-HyperNEAT results cited by the authors reach 29% on MNIST, while spatially specialized experts reach 49%.Source: Limitations · p. 8
  • Feedforward results concern evolved substrates with few hidden nodes; monotonic functions can represent parity with sufficient width, but evolution did not discover that construction.The construction requires width Θ(n).Source: Limitations · p. 8
  • The Hanoi plateau near 78% and Two Spirals ceiling near 77% persist in feedforward and recurrent architectures, indicating mismatches topology alone cannot resolve.Source: Limitations · p. 8
  • The biological parallel is that distinct neuron types meet different computational demands; whether evolved assignments form spatially coherent regions remains future work.Source: Limitations · p. 8

Conclusion

The paper reports categorical solvability differences among activation functions, alongside an assignment mechanism that outperforms uniform and weight-derived alternatives. These differences depend on the problem, architecture, and optimizer, while indirect encoding lets evolution search per-node assignments impractical to choose by hand.

RQ1: Categorical solvability limits

  • Oscillatory functions solve parity at 100% where monotonic functions reach 0%; sparse neuroevolution did not discover the width Θ(n) monotonic construction, though one oscillatory unit suffices.The 18-function divide is predicted by oscillatory structure (ρ=0.81) and holds across population sizes 50–2,000.Source: Conclusion · p. 8

RQ2: Assignment mechanism

  • CPPN-controlled per-node assignment significantly outperforms weight-interpretation and uniform methods.Source: Conclusion · p. 8

RQ3: Boundary conditions

  • The divide is absent on sine regression, present on concentric circles, collapsed by recurrence on parity, and inverted by gradient descent, where tanh solves through Parity-8 while sin falls to 0%.Source: Conclusion · p. 8
  • Indirect encoding lets evolution search per-node activation assignments impractical to choose by hand, shaping substrates through function category, problem geometry, and optimizer.Source: Conclusion · p. 8
Loading 2610.00149v1…