Fractional Optimizers Meet Fractal Activation Functions: An Empirical Study of Multi-Scale Optimization in Neural Network

• Fractal activations and fractional optimizers are studied together because both address multi-scale structure, respectively through representation roughness and memory in the update direction.The paper frames their relationship through the shared Grünwald–Letnikov mechanism linking fractal derivatives and optimizer memory.Source: §1 · pp. 2, 4
• The study evaluates optimizer behavior on Ackley and Himmelblau surfaces, with and without additive Weierstrass-type perturbations, and in networks using fractal or conventional activations across ten classification datasets.These experiments assess controlled fractal landscapes and neural-network classification performance within one framework.Source: §1 · pp. 2–3
• The unified analysis connects fractional derivatives of Weierstrass-type functions with discrete Grünwald–Letnikov optimizer constructions, explaining fractal activations and fractional optimizers as realizations of one mechanism.The connection spans continuous fractional-derivative theory, discrete memory constructions, and their role in automatic differentiation.Source: §1 · pp. 2–3
• The paper introduces AdaptiveMemoryFSGD, -FRMSprop, -FAdam, and -FAdadelta, combining fractional-memory and ordinary gradients through a stability- and loss-controlled mixing coefficient.The adaptive framework fixes the fractional order, bounds the memory contribution, and exactly recovers the underlying classical optimizer at one coefficient boundary.Source: §1 · p. 3
• The empirical program compares 21 optimizers across classical, Herrera-type fractional, memory-based fractional, adaptive memory-based fractional, and related-work groups.The comparison proceeds from fractally perturbed benchmark surfaces with known minima to neural-network classification using fractal and conventional activations.Source: §1 · pp. 3–4

Fractal functions in classical analysis

• Weierstrass- and Blancmange-type series provide the raw material for both fractal benchmark perturbations and neural-network activation functions.The study builds on classical constructions including the Weierstrass function, Takagi function, and Weierstrass–Mandelbrot function.Source: §2 · pp. 4, 6

Fractional derivatives of Weierstrass-type functions

• Fractional calculus links rough fractal functions to nonlocal optimization, with Grünwald–Letnikov derivatives providing a discrete representation for gradient-history methods.For Weierstrass-type functions, fractional derivatives can exist below the roughness exponent γH even when classical derivatives fail.Source: §2 · pp. 4–6

Fractality in neural networks

• Fractality in neural networks spans training dynamics, architectures, features, interpolation, and activation nonlinearities, with providing the direct predecessor for this study.The predecessor established stable Weierstrass- and Blancmange-based activations, evaluated them on ten datasets, and analyzed expressivity and gradient stability.Source: §2 · p. 5

Fractional optimizers for neural networks

• Fractional neural-network optimizers comprise power-law gradient rescaling, explicit Grünwald–Letnikov history, and adaptive-order designs, alongside hybrid methods such as AdaGL.The present adaptive memory-fractional framework fixes the fractional order and bounds memory through a hysteresis-controlled trust coefficient.Source: §2 · pp. 5–6

Other fractional optimization approaches

• Other fractional optimization methods apply generalized derivatives, adaptive schemes, accumulation-level weighting, classifier-specific training, and application-oriented learning beyond the optimizer families studied here.Examples include ψ-Hilfer methods, stochastic fractional gradients, fractional SVM training, FracGrad, and tempered meta-learning.Source: §2 · p. 6

Positioning of the present study

• The study jointly evaluates fractional optimizer families on controlled Weierstrass-perturbed landscapes and with fractal activations across 21 optimizers under identical classification conditions.The authors identify no prior work combining these evaluations in one framework.Source: §2 · p. 6

3.1. Fractals and Fractal Functions

• Fractals repeat simple rules across scales, while Weierstrass-type functions transfer self-similarity to curves through decreasing-amplitude, increasing-frequency oscillations.Finite Weierstrass sums are ordinary functions, but adding scales produces increasingly rough behavior and can approach nowhere differentiability in suitable parameter ranges.Source: §3.1 · pp. 7–8
• Fractal activations add localized, alternating oscillatory scales to a smooth tanh backbone, with depth M increasing the number of represented scales.The amplitude factor decreases higher-scale contributions, the frequency factor increases them, and the exponential term localizes the correction around the origin.Source: §3.1 · p. 9

3.2. Fractional Derivatives

• Fractional derivatives generalize differentiation to noninteger orders by combining information over an interval or history, with Grünwald–Letnikov coefficients giving nonzero weights to past values [11, 12, 13].A derivative of order 3/2 lies between first- and second-order differentiation, while the discrete Grünwald–Letnikov form supports iterative optimization through finite-history fractional gradients.Source: §3.2 · pp. 9–10

3.3. Fractional Derivatives of Fractal Functions

• When ab > 1, higher-frequency terms can make ordinary derivatives large and strongly oscillatory even as the original finite Weierstrass function remains bounded.A finite-memory Grünwald–Letnikov derivative retains local structure while incorporating weighted history, motivating fractional optimizer updates that respond to previous gradients rather than only the current one.Source: §3.3 · p. 11

4. Weierstrass-Type Functions and Fractional Derivatives

• Weierstrass-type functions and fractional derivatives share multiscale structure, making fractional differentiation a natural tool for measuring roughness across scales.Weierstrass constructions generate benchmark perturbations and network activations, while fractional derivatives generate memory-based optimizers.Source: §4 · p. 12

4.1. Weierstrass-Type Functions and Their Fractional Differentiability

• For roughness exponent γH, Weyl–Marchaud derivatives exist for ν < γH, reduce roughness to γH −ν, measure roughness at ν = γH, and generally fail above it.At the critical order, the signed gradual derivative vanishes almost everywhere while the absolute version equals a positive constant; Weyl–Marchaud derivatives are chosen for increment-based, periodic, self-similar functions.Source: §4.1 · pp. 14–15

4.2. The Discrete Grünwald–Letnikov Formulation and Automatic Differentiation

• The discrete Grünwald–Letnikov operator converts fractional calculus into a power-law weighted history sum, with ν controlling memory and ν = 1 recovering the ordinary backward difference.Truncating the history to window K creates a finite-memory approximation that trades computational cost and numerical sensitivity against fidelity to the full fractional operator.Source: §4.2 · pp. 16–17
• Fractional activations modify pointwise feature-space nonlinearities, whereas fractional optimizers aggregate past gradients in parameter space; automatic differentiation applies the ordinary chain rule to both implemented discrete programs.Grid resolution can hide or misrepresent fine Weierstrass scales, and memory window K also determines the optimizer’s computational budget.Source: §4.2 · pp. 17–18
• Discrete Grünwald–Letnikov differentiation represents roughness through a history-weighted sum whose memory length K controls computational cost and retained history.At ν = 1, the coefficient structure recovers an ordinary difference, while K = 1 yields the classical gradient-descent fallback; short histories can suffice in practice.Source: §4.2 · p. 19, §4.3 · pp. 20, 23

4.3. Fractional Gradient Descent With and Without Memory for a Weierstrass-Based Activation

• The modified Weierstrass–Tanh activation from Raubitzek et al. is selected because it is reliable across datasets and milder than extreme Weierstrass-based variants.Its truncated implementation uses a = 0.5, b = 1.5, µe = 0.75, and N = 100, with tanh providing a stable backbone and the envelope localizing oscillations.Source: §4.3 · p. 19
• Backpropagation remains ordinary: automatic differentiation computes gradients through the truncated activation, and fractionalization enters only in the parameter update.The hereditary branch combines recent gradients with a power-law kernel, while the non-hereditary branch rescales the current step without storing history.Source: §4.3 · pp. 20–22
• The activation shapes fine-scale gradient oscillations, while the optimizer determines whether those patterns persist across iterations; memory helps when oscillations are structured but hurts when they are noise-like.The order ν and window K act as regularity controls: smaller ν gives older gradients more influence, whereas larger ν makes updates more local.Source: §4.3 · pp. 21, 23
• For bounded, moderately irregular Weierstrass–Tanh activations with ab < 1, hereditary fractional updates preserve useful gradient memory, whereas memoryless scaling offers a lighter fractional surrogate.Aggressive variants may require shorter memory, clipping, normalization, or norm matching because long memory can amplify oscillations.Source: §4.3 · pp. 22–23

5.2. The Four Activation Functions Used in This Study

• The study selects four representative activations—modulated Blancmange, decaying cosine, modified Weierstrass–Tanh, and Weierstrass–Mandelbrot x+ sin—to keep the optimizer–activation grid tractable.The subset includes the predecessor study’s most reliable performer, dataset-specific variants, and different levels of multiscale irregularity.Source: §5 · p. 23, §5.1 · p. 24

5.2.1. Modulated Blancmange Curve (modulated_blancmange_curve)

• The modulated Blancmange curve uses a critical a = 1/2, b = 2 ladder whose finite implementation produces bounded, frequently changing gradient signals across scales.Its input-dependent offset and tanh gate modify the standard Takagi construction, while retained corner spacings are of order 2^-m.Source: §5.2.1 · p. 25

5.2.2. Decaying Cosine Function (decaying_cosine_function)

• The decaying cosine uses a critical cosine ladder with a smooth tanh(πx) backbone, an e^-|x|/2 envelope, and odd-sign modulation that localizes its oscillatory correction.Its normalization ζ = 0.2666 with N = 75 makes the repeated backbone approximately unit-scale tanh(πx), and it was among the more stable activations in.Source: §5.2.2 · pp. 25–26

5.2.3. Modified Weierstrass–Tanh (modified_weierstrass_tanh)

• The modified Weierstrass–Tanh is subcritical because ab = 0.75 < 1 and γH ≈ 1.71, giving a rapidly varying but bounded derivative ladder away from its envelope cusp.Its tanh backbone, alternating-scale cancellation, and localized envelope define the balanced regime identified in, making it the reference case for optimizer-level analysis.Source: §5.2.3 · p. 26

5.2.4. Weierstrass–Mandelbrot (weierstrass_mandelbrot_xpsin)

• The four activations deliberately span subcritical, critical, and supercritical ladders, with Weierstrass–Mandelbrot x+ sin providing the roughest case: γH = 1/2 and a b = 2 > 1.Its linear drift preserves average gradient propagation when oscillations cancel, while the selected regime range supplies controlled multiscale gradient roughness for fractional-optimizer evaluation.Source: §5.2.4 · p. 27, §5.3 · pp. 27–28, Table 1 · p. 27

5.1. The General Construction

• Fractal activations combine a stable backbone with shrinking-amplitude, increasing-frequency oscillations, while truncation yields finite tensor expressions compatible with automatic differentiation.The retained scales approximate the numerically representable portion of the ideal series; harmonics beyond approximately N = 30–100 are near machine precision for the chosen parameters.Source: §5.1 · p. 24, §5.3 · p. 27

6.1. Classical First-Order Baselines: SGD, RMSprop, Adam, Adadelta

• Classical SGD, RMSprop, Adam, and Adadelta provide first-order, short-memory baselines whose state uses at most two exponentially weighted moments.Fractional variants modify these base update rules while preserving their optimizer-specific state mechanisms.Source: §6 · p. 29, §6.1 · pp. 29–30

6. Fractional Optimizers

• The comparison contains 21 optimizers in five groups, with fractional-order sweeps limited to Herrera-type and memory-based methods while adaptive and related-work methods use fixed internal configurations.The swept orders are ν ∈ {0.75, 1.25, 1.50}; displayed ν = 1.00 entries for adaptive or related-work methods are placeholders, not actual internal orders.Source: §6 · p. 29

6.2. From Integer-Order to Fractional-Order Updates

• Fractional optimization replaces the first-order gradient through either local Caputo-inspired rescaling of the current gradient or nonlocal Grünwald–Letnikov combination of recent gradients.Local rescaling stores no history, whereas the memory route uses an algebraically decaying kernel over a finite gradient history.Source: §6.2 · pp. 30–31

6.3. Herrera-Type Fractional Gradient-Scaled Optimizers

• Herrera-type optimizers replace the current gradient input with a Caputo-inspired, componentwise power-law scaling while storing no gradient history.They preserve the gradient direction, modulate componentwise magnitudes, and reduce exactly to their base optimizers at ν = 1.Source: §6.3 · pp. 31–32

6.4. Memory-Based Fractional Optimizers

• Memory-based optimizers construct a finite-history Grünwald–Letnikov direction, then apply descent safeguarding and norm matching to prevent cancellation or uphill updates.The memory direction replaces the numerator or directional component, while raw gradients drive squared-gradient accumulators; neural-network experiments use history length six.Source: §6.4 · pp. 33–34

6.5. Adaptive Memory-Based Fractional Optimizers

• Adaptive memory optimizers keep ν fixed at 1.25 and adapt a bounded mixing coefficient λ_t between the ordinary and memory-based gradients.They use K = 4, initialize λ_0 = 0.10, clip λ_t to [0, 0.30], and therefore limit the memory contribution to 30%.Source: §6.5 · pp. 34–35
• Adaptive-memory optimizers use λt to mix ordinary and norm-matched fractional gradients, increasing memory when gradients are stable and withdrawing it when loss worsens.They recover the classical optimizer exactly at λt = 0; with λmax = 0.30, at least 70% of each update direction remains ordinary gradient.Source: §6.5 · p. 36, §6.7 · p. 40

6.6. Related Fractional Optimizers

• Related methods introduce fractional structure through permanent Grünwald–Letnikov substitution, always-fractional step rescaling, or adaptive-order local scaling, without the adaptive framework’s exact classical fallback.FCSGD_GL and FCAdam_GL use Bernoulli-perturbed fixed-order history, AdaGL adds a short-term multiplier, and AOFGD adapts νt.Source: §6.6 · pp. 37–38

6.7. Summary of the 21 Optimizers

• The comparison contains 21 optimizers in five groups, separating local scaling from explicit memory, fixed from adaptive control, and permanent substitution from safeguarded use with exact fallback.Memory methods share a common kernel, while safeguards, norm matching, and λt regulate history’s influence and permit recovery of each base optimizer.Source: Table 2 · p. 39, §6.7 · pp. 39–40

7.1. Benchmark Surfaces, Fractal Perturbations, and Experimental Design

• The experiments compared 21 optimizer families on standard and additively fractal-perturbed Ackley and Himmelblau surfaces, using 40 repeated runs and shared controlled initialization protocols.They recorded final and best loss, success rate, distance to the nearest known minimum, optimization steps, and runtime; perturbations preserved global basins while adding multi-scale local roughness.Source: §7.1 · pp. 40–41, §7.2 · p. 43

7.2. Surface Optimization Results

• FCSGD_GL solved standard Himmelblau completely, AOFGD_Adam led additive Himmelblau with 0.73 success and 1.98 mean final loss, while Ackley leaders remained at 4.6–5.8 loss.On standard Ackley, FSGD led with 0.15 success; on additive Ackley, FCSGD_GL had 4.59 mean final loss and MemoryFSGD the highest success rate, 0.13.Source: §7.2 · p. 43, Table 3 · p. 43
• Adadelta-based memory variants sometimes diverged on Himmelblau, producing mean final losses around 10^16 despite moderate medians, so distributions and success rates were essential.Runtime also increased with fractional, memory, and adaptive updates: FCSGD_GL required roughly 1.9–2.8 times SGD’s standard-surface step time, while perturbation evaluation added about 5–6 ms per step to all methods.Source: §7.2 · pp. 47–48

7.3. Discussion and Summary

• Because these are artificial two-dimensional surfaces with fixed budgets and initialization, the findings provide mechanistic optimizer diagnostics rather than direct evidence of neural-network training performance.Known minima make distance interpretable, but later classification experiments are required to test practical advantages on learned models.Source: §7.3 · pp. 48–49
• Perturbations reduced reliability but preserved broad family rankings on Himmelblau, whereas Ackley’s altered objective values did not overcome its plateau; SGD-type methods remained the only consistent achievers.RMSprop- and Adadelta-based methods were weak across configurations, while fractional and memory-based SGD variants matched or exceeded classical SGD counterparts.Source: §7.2 · pp. 44, 48, §7.3 · p. 49

8.1. Experimental Pipeline, Architectures, Activations, and Optimizers

• The benchmark covers ten OpenML datasets spanning binary and multi-class classification, providing a varied testbed for optimizer–activation interactions.The datasets are climate-model-simulationcrashes, diabetes, ionosphere, tic-tac-toe, vertebra-column, glass, iris, seeds, vehicle, and wine.Source: §8.1 · p. 50
• The experiment compares ReLU and tanh with four selected fractal activations and 21 optimizers organized into five conventional, fractional, memory-based, adaptive-memory, and related-work groups.The selected fractal activations are the modulated Blancmange curve, decaying cosine, modified Weierstrass–tanh, and Weierstrass–Mandelbrot x + sin.Source: §8.1 · pp. 50–51

8.2. Classification Results

• Optimizer rankings use each optimizer’s highest-mean-accuracy configuration, whereas activation figures average performance across all optimizer and derivative-order settings.This separates best available optimizer configurations from overall activation behavior across the experimental grid.Source: §8.2 · p. 51

8.2.1. Climate-Model Simulation Crashes

• On Climate-Model Simulation Crashes, Table 7 presents the top 15 optimizer-specific configurations, ranked by mean test accuracy over 40 repeated runs, showing each optimizer’s highest-ranked activation–order configuration.Standard deviation quantifies variation across repeated runs.Source: Table 7 · p. 52

1 AdaptiveMemory FAdadelta

• 0.91250 ± 0.01740 was the highest Climate-Model Crashes accuracy, achieved by AdaptiveMemoryFAdadelta with modified Weierstrass–tanh at ν = 1.00.Standard Adadelta and MemoryFSGD both reached 0.91204 ± 0.01789, only 0.046 percentage points lower, with overlapping confidence intervals; the complete optimizer range was 0.91250 to 0.90586.Source: §1 · pp. 52–53

8.2.2. Diabetes

• On Diabetes, FSGD with ReLU and ν = 1.25 led at 0.77197 ± 0.02215, while the best fractal configuration, AdaptiveMemoryFAdam with modified Weierstrass–tanh, reached 0.76981 ± 0.02222.The 0.216-percentage-point gap was small relative to approximately 2.2-percentage-point standard deviations, and confidence intervals overlapped; FSGD also required 1.36 s versus 7.59 s for AdaptiveMemoryFAdam.Source: §8.2.2 · p. 55, Table 8 · p. 55
• Diabetes results did not support a general explicit GL-memory advantage: MemoryFSGD reached 0.76158 ± 0.02585, while MemoryFAdam, MemoryFRMSprop, and MemoryFAdadelta were lower.Across all 21 optimizers, accuracy ranged from 0.77197 for FSGD to 0.68517 for AdaGL, indicating method-specific rather than family-wide differences.Source: §8.2.2 · pp. 56–57
• Modified Weierstrass–tanh was the strongest aggregated activation on Diabetes at approximately 0.755, exceeding tanh at approximately 0.732, whereas the fractal-family mean was lower than the standard-family mean.The standard-family mean was 0.72942 versus 0.69828 for fractal activations because the latter included weaker variants, especially Weierstrass–Mandelbrot x + sin.Source: §8.2.2 · p. 57

8.2.3. Glass Identification

• Glass Identification comprised 222 optimizer–activation–order configurations evaluated over 40 repeated runs, with Table 9 reporting each optimizer’s best activation–order configuration.The dataset contains 214 samples, nine continuous measurements, and six glass-type classes.Source: §8.2.3 · p. 58, Table 9 · p. 58

1 AdaptiveMemory FAdadelta

• On Glass, AdaptiveMemoryFAdadelta with decaying cosine and ν = 1.00 ranked first at 0.59538 ± 0.06739, but its advantage over faster FAdadelta–tanh was small relative to run variation.The best explicit GL-Memory method, MemoryFAdam, reached only 0.50885 ± 0.06741, while tanh led the aggregated activation ranking at approximately 0.457.Source: §1 · pp. 58–60

8.2.4. Ionosphere

• On Ionosphere, FSGD with modified Weierstrass–tanh at ν = 1.50 ranked first at 0.91745 ± 0.03502, while explicit GL-Memory methods occupied the lower ranking.The top configuration’s confidence interval overlapped those of following methods, and the best GL-Memory optimizer, MemoryFSGD, reached 0.88703 ± 0.03728.Source: Table 10 · p. 61, §8.2.4 · pp. 61–63, Figure 22 · p. 62

Activation-level comparisons

• Activation results varied by dataset: modified Weierstrass–tanh led Ionosphere aggregation, decaying cosine led Iris aggregation, and tanh led Glass aggregation despite selected fractal pairings topping configurations.On Ionosphere, modified Weierstrass–tanh averaged approximately 0.869; on Iris, decaying cosine averaged approximately 0.779; on Glass, tanh averaged approximately 0.457.Source: §1 · p. 60, §8.2.4 · p. 63, §8.2.5 · p. 66

8.2.5. Iris

• On Iris, FAdam with decaying cosine at ν = 1.25 led at 0.93444 ± 0.03374, while explicit GL-Memory methods again performed substantially worse.Decaying cosine also achieved the highest aggregated activation accuracy at approximately 0.779, although Standard and Herrera groups had nearly identical aggregate accuracy.Source: Table 11 · p. 64, §8.2.5 · pp. 64–66, Figure 24 · p. 65

8.2.6. Seeds

• The Seeds subsection describes a 210-sample, three-class dataset evaluated across 222 optimizer–activation–order configurations and 8,880 repeated runs, with Table 12 retaining each optimizer’s best setting.No Seeds performance results are supplied in this section part.Source: §8.2.6 · p. 67
• FSGD with tanh and ν = 0.75 led Seeds at 0.91230 ± 0.04222, while AdaptiveMemoryFSGD ranked second at 0.90595 ± 0.04104 and required 9.91 s versus 1.29 s.The 0.635-percentage-point gap was small relative to run-to-run variation, and confidence intervals overlapped.Source: §8.2.6 · pp. 67–68, Table 12 · p. 67

Cross-dataset comparison

• Explicit GL-Memory methods did not enter the top 15 on Seeds, Vertebral Column, or Wine Recognition and generally occupied lower rankings with greater variation.MemoryFAdadelta reached 0.73254 on Seeds, 0.75860 on Vertebral Column, and 0.90324 on Wine Recognition.Source: §8.2.6 · pp. 68–69, §8.2.7 · pp. 71–72, §8.2.8 · p. 75

8.2.6. Seeds

• On Seeds, Herrera optimizers had the highest aggregate accuracy at 0.77962, while ReLU led activations at approximately 0.769 and decaying cosine was the strongest fractal activation at approximately 0.754.Standard activations averaged 0.74617 versus 0.67888 for fractal activations, so the dataset showed no general fractal-activation advantage.Source: §8.2.6 · pp. 69–70, Figure 27 · p. 69

Cross-dataset comparison

• The results show optimizer–activation interaction: standard activations dominated aggregate performance, but fractal functions produced leading configurations on Wine Recognition and competitive selected pairings elsewhere.On Seeds, the overall winner used tanh, whereas Wine Recognition’s winner used modulated Blancmange.Source: §8.2.6 · p. 70, §8.2.7 · p. 73, §8.2.8 · pp. 74, 76

8.2.7. Vertebral Column

• FAdam with tanh and ν = 1.50 led Vertebral Column at 0.83763 ± 0.03527, followed by FAdadelta at 0.83360 ± 0.02932; their confidence intervals overlapped.FAdam also led macro-F1 at 0.79949, with 1.81 s mean training time versus 1.65 s for FAdadelta.Source: §8.2.7 · pp. 70–71, Table 13 · p. 70
• Vertebral Column favored Herrera in aggregate accuracy at 0.72565, while tanh led activations at approximately 0.773 and standard activations averaged 0.75078 versus 0.58860 for fractal activations.Modified Weierstrass–tanh remained competitive in selected pairings, but explicit GL-Memory methods remained below direct fractional and standard alternatives.Source: §8.2.7 · pp. 71–73, Figure 29 · p. 72

8.2.8. Wine Recognition

• AOFGD_Adam with modulated Blancmange led Wine Recognition at 0.96111 ± 0.02807, ahead of Adadelta at 0.95972 ± 0.02810; the 0.139-point gap had overlapping intervals.AOFGD_Adam led macro-F1 at 0.96231 but required 7.19 s versus 5.44 s for Adadelta.Source: §8.2.8 · pp. 74–75, Table 14 · p. 74
• Wine Recognition’s top 15 all used fractal activations, yet tanh led the full activation grid at approximately 0.857 while modulated Blancmange ranked fifth overall.Standard activations averaged 0.81398 versus 0.69657 for fractal activations, showing strong selected pairings but weaker average robustness.Source: §8.2.8 · pp. 74, 76, Figure 31 · p. 76
• On Wine Recognition, modulated Blancmange with AOFGD_Adam at ν = 1.00 produced the highest-ranked configuration, but tanh had the strongest activation-wide mean and Standard the highest optimizer aggregate.The result demonstrates dataset-specific tuning: a fractal activation achieved the best configuration without dominating performance across the full optimizer grid.Source: §8.2.8 · pp. 76–77

8.2.9. Tic-Tac-Toe Endgame

• The clearest optimizer–activation interaction occurred on Tic-Tac-Toe Endgame, where FSGD with modified Weierstrass–tanh at ν = 1.50 reached 0.78411 ± 0.02705, ahead of AdaptiveMemoryFSGD at 0.76337 ± 0.02729.Modified Weierstrass–tanh appeared in all top-15 configurations and averaged approximately 0.730, although standard activations averaged 0.71260 versus 0.67380 for the fractal family.Source: §8.2.9 · pp. 77–80, Table 15 · p. 77

8.2.10. Vehicle Silhouettes

• On Vehicle Silhouettes, FAdadelta with ReLU and AOFGD_Adam with ReLU tied at mean accuracy 0.79163, while explicit GL-Memory methods ranked lower and all top-15 configurations used ReLU.Standard activations averaged 0.63741 accuracy versus 0.48916 for fractal activations.Source: §8.2.10 · pp. 81–83

8.2.11. Summary of the Classification Results

• Across ten datasets, fractal activations appeared in most top-15 configurations on eight datasets and won six datasets, while activation-wide means favored fractal functions on only four.Modified Weierstrass–tanh was the most consistent fractal activation, but results varied substantially among functions; Herrera-type optimizers supplied the sole top result on six datasets, while explicit GL-memory methods produced none.Source: §8.2.11 · pp. 84–85
• The broader experiments show that fractional optimization also helped with standard activations, including FSGD with ReLU on Diabetes and FAdadelta with ReLU on Vehicle Silhouettes, rather than only supporting fractal activations.The strongest interaction between fractional optimization and fractal activation occurred on Ionosphere, Iris, and Tic-Tac-Toe Endgame; elsewhere, effects depended on the dataset and activation.Source: §8.2.11 · pp. 84–85
• Explicit GL-Memory optimizers usually ranked outside the top 15, showed larger variation, and averaged below direct Herrera, Standard, and AdaptiveMemory families on most datasets.Their weaker results may partly reflect sensitivity to fractional order, since no single tested order was consistently optimal; this remains a plausible rather than confirmed cause.Source: §8.2.11 · pp. 85–87
• Fractal activations dominated optimizer-specific rankings on eight of ten datasets, but activation-wide averages showed their improvement was not uniform.Standard activations were more robust on several datasets, supporting joint rather than independent method selection.Source: §8.2.11 · pp. 86–87
• Fractal activations formed most top-15 results on eight datasets and provided the best configuration on six, while standard activations led on Vertebral Column and Vehicle Silhouettes.The strongest pairings included modified Weierstrass–tanh on Ionosphere, Diabetes, and Tic-Tac-Toe; decaying cosine on Iris and Glass; and Blancmange on Wine.Source: §8.2.11 · p. 87
• Direct fractional optimizers remained consistently competitive, with Herrera-type methods leading on six datasets and sharing first place on Vehicle.These leaders used both fractal and standard activations, indicating that optimizer–activation interactions shaped the strongest results.Source: §8.2.11 · p. 87
• Across 222 configurations, 40 runs, and ten datasets, overlapping error ranges made single-run rankings unreliable and favored interpreting repeated effective pairings over universal winners.On most datasets, leading mean accuracies differed little relative to 40-run variability; Tic-Tac-Toe provided a clearer separation, with FSGD plus modified Weierstrass–tanh and ν = 1.50 exceeding following configurations by more than two percentage points.Source: §8.2.11 · pp. 86–87

• SGD-type methods dominate Ackley and Himmelblau variants, with fractional or memory-based SGD matching or exceeding plain SGD while perturbations alter rankings.FSGD leads both Ackley variants, FCSGD_GL exactly solves standard Himmelblau, and MemoryFSGD has the highest additive-Ackley success rate; AOFGD_Adam leads perturbed Himmelblau.Source: §9 · pp. 87, 88, 92
• Fractal activations win through optimizer pairings: they dominate top-15 configurations on eight datasets and win six, but activation-wide averages favor them on only four.Modified Weierstrass–tanh is most consistent, whereas the roughest Weierstrass–Mandelbrot x+ sin activation generally has the lowest activation-wide means.Source: §9 · pp. 88, 92
• Herrera-type local scaling is the strongest optimizer design, taking six dataset wins and one shared win, while no fractional order is universally optimal.Winning configurations use ν = 0.75, 1.25, and 1.50; explicit GL-memory methods are usually absent from top-15 results outside Climate and show greater variation.Source: §9 · pp. 88, 89, 92
• Explicit gradient memory helps deterministic multi-scale surfaces but amplifies minibatch noise, whereas the bounded λt framework ranks above plain GL memory and wins two datasets.AdaptiveMemoryFSGD is reliable across surface variants, while its additive-Himmelblau failure is inherited from Adadelta; λt provides an exact classical fallback.Source: §9 · pp. 89, 90, 92
• Jointly selecting activation, optimizer, and order is the practical consequence, with Herrera methods combining high accuracy and low cost while adaptive memory costs more.Herrera training averages 6.5 s, FAdam 5.5 s, and adaptive memory 9.9 s; most leading accuracies differ by less than run-to-run variation, and conclusions remain limited to feed-forward networks and ten tabular datasets.Source: §9 · pp. 90–92
• Specific pairings, especially subcritical fractal activations with local-scaling or adaptive fractional optimizers, repeatedly outperform indiscriminate explicit gradient memory.The two-stage design separates memory behavior on controlled multi-scale landscapes from behavior under stochastic network gradients.Source: §9 · p. 91

Main findings

• SGD-type methods led every controlled-surface variant, while Herrera-type scaling won six of ten classification datasets and fractal activations won six, leading configurations on eight.Explicit gradient memory was reliable on perturbed landscapes, whereas classification favored local scaling; fractal-activation gains depended on specific optimizer pairings rather than grid-wide averages.Source: §10 · pp. 92–93

Robustness and scope

• No universally optimal fractional order emerged, although ν ∈{0.75, 1.25, 1.50} covered all winning configurations and most leading differences were small relative to run-to-run variation.The study evaluated 21 optimizers across five families, with 40 repeated runs for controlled surfaces and classification experiments.Source: §10 · pp. 92–93

Memory mechanisms

• The adaptive memory framework ranked above plain memory methods throughout and won two datasets by bounding memory with a trust coefficient λ_t and retaining an exact classical fallback.On fixed multi-scale surfaces, truncated Grünwald–Letnikov memory extracts persistent oscillatory structure; under stochastic minibatches, its differencing amplifies noise, while safeguards preserve functionality.Source: §10 · p. 93

Practical recommendations

• Practitioners should select activation, optimizer formulation, and fractional order jointly; use Herrera-type scaling by default, treat order as a small discrete hyperparameter, and reserve explicit memory for low-noise settings.Bounded adaptive mixing is recommended when explicit memory is applied outside low-noise settings with persistent multi-scale structure.Source: §10 · p. 93

Limitations and future work

• The study provides a broad, coherent foundation across perturbed benchmark surfaces and tabular classification, while convolutional, recurrent, transformer, graph, regression, generative, and larger-benchmark evaluations remain open.Future methodological work includes memory length, adaptation hyperparameters, trust-coefficient schedules, and combining order adaptation with trust adaptation.Source: §10 · p. 93

Overall contribution

• Fractal activations and fractional optimizers are mathematically compatible modifications whose benefits are identifiable in subcritical activation–local-scaling or adaptive-optimizer pairings, not through indiscriminate gradient memory.The unified framework, safeguarded implementations, and repeated-run protocol provide a reproducible basis for further mathematically structured deep-learning methods.Source: §10 · p. 94

Funding

• SBA Research’s COMET Center is funded by BMIMI, BMWET, and Vienna, with additional support acknowledged from Austrian federal, national research, and Christian Doppler organizations.The COMET Programme is managed by the Austrian Research Promotion Agency (FFG).Source: Funding · p. 94

Abbreviations; Appendix A. Notation; Appendix A.1. General Conventions; Appendix A.2. Fractional Order and Grünwald–Letnikov Coefficients; Appendix A.3. Adaptive Mixing and Related-Work Control Quantities; Appendix A.4. Loss-Aware Control; Appendix A.5. Base-Optimizer State Variables; Appendix A.6. Weierstrass-Type Functions and Continuous Fractional Operators; Appendix A.7. Discrete Fractional Operators and Network-Layer Notation; Appendix A.8. Fractal Activation Functions and Single-Neuron Notation

• The appendix defines abbreviations and notation for fractional orders, finite-memory gradients, adaptive mixing, loss-aware controls, optimizer states, fractal functions, and network-layer quantities.The notation glossary specifies scalar and vector conventions, iteration indices, componentwise powers, and the tables organizing these symbols.Source: Abbreviations · pp. 96–97, Appendix A · p. 98

Appendix B. Extensions of the Memory-Based Optimizer Families

• The appendix extends fixed and adaptive memory mechanisms from SGD, RMSprop, Adam, and Adadelta to Adagrad, AdamW, Nadam, and Adamax.The extensions use the same notation and safeguards as the main constructions and are intended to show that the mechanisms are not restricted to the four main optimizer families.Source: Appendix B · p. 104

Appendix B.1. Fixed Memory-Based Extensions

• Fixed-memory variants apply a safeguarded, norm-matched finite-history Grünwald–Letnikov direction while keeping squared-gradient and coordinatewise scale accumulators based on the raw gradient.At ν = 1 or history length one, the explicit fallback reduces each method to its corresponding base optimizer.Source: Appendix B.1 · pp. 104–105
• MemoryFAdagrad retains Adagrad’s monotone coordinatewise accumulator but replaces its numerator with a safeguarded fractional-memory direction.It therefore combines decreasing effective learning rates with a search direction that depends on short gradient history.Source: Appendix B.1 · p. 104
• MemoryFAdamW, MemoryFNadam, and MemoryFAdamax apply fractional-memory directions to their Adam-family updates while preserving decoupled decay, Nesterov structure, or raw-gradient infinity-norm scaling.For MemoryFNadam, fractional memory enters both the first moment and look-ahead term, while the second moment remains raw-gradient based.Source: Appendix B.1 · pp. 104–105

Appendix B.2. Adaptive Memory-Based Extensions

• Adaptive variants use the same fractional gradient, safeguards, and stability- and loss-aware mixing coefficient λt, while raw gradients remain the default source for scale accumulators.λt = 0 gives the exact base optimizer, and ν = 1 does so under the classical fallback convention.Source: Appendix B.2 · p. 105
• AdaptiveMemoryFAdagrad increases memory-direction contribution when recent gradients are stable and decreases it when gradients become inconsistent or smoothed loss worsens.Unlike MemoryFAdagrad, its memory contribution is not fixed.Source: Appendix B.2 · pp. 105–106
• AdaptiveMemoryFAdamW, AdaptiveMemoryFNadam, and AdaptiveMemoryFAdamax control memory-aware directions through λt while retaining independent weight decay, Nadam fallback behavior, or raw-gradient infinity-norm scaling.AdaptiveMemoryFNadam is exactly Nadam when λt = 0 or ν = 1, and AdaptiveMemoryFAdamax can reduce memory contribution during unstable phases.Source: Appendix B.2 · p. 106

Appendix B.3. Summary of the Optimizer-Family Extensions

• Fixed-memory variants use the safeguarded, norm-matched Grünwald–Letnikov gradient as the update direction while estimating scales from the raw gradient.At ν = 1, each extension recovers its corresponding base optimizer.Source: Appendix B.3 · p. 107, Table B.25 · p. 107
• The extensions transfer fixed- and adaptive-memory principles to additional first-order optimizers beyond the 21 methods in the main comparison.Table B.25 summarizes these extensions rather than adding them to the main experimental tables.Source: Table B.25 · p. 107, Appendix B.3 · p. 107
• Adaptive variants update with (1 − λ_t)g_t + λ_tĝ_t^(ν), retain raw-gradient scale estimates, and recover base optimizers at λ_t = 0 or ν = 1.The adaptive coefficient λ_t is bounded.Source: Appendix B.3 · p. 107
• The memory mechanisms are therefore not tied to the four main-experiment families: the safeguarded fractional-memory direction replaces the base optimizer’s descent direction while scales remain raw-gradient based.This transfer rule applies unless scale estimation is explicitly specified otherwise.Source: Appendix B.3 · p. 107

Appendix C. Runtimes of the Experiments

• Runtime measurements make explicit the computation accompanying accuracy and success-rate differences across both experimental blocks.The appendix treats computational cost as part of interpreting the main experimental comparisons.Source: Appendix C · p. 107
• Surface experiments report mean wall-clock time per optimization step, isolating update-rule cost because optimizers use identical step counts on the same objective.This quantity is specific to the surface block.Source: Appendix C · p. 107
• Neural-network experiments report each optimizer’s best-configuration mean training time per dataset, averaged across ten datasets.This cost accompanies the accuracy values reported in Section 8.Source: Appendix C · p. 108
• All runtime values average 40 repeated runs per configuration on common hardware within each block, supporting within-table comparisons but not cross-table or cross-hardware transfer.Comparability is limited to measurements within the same experimental block and hardware setting.Source: Appendix C · p. 108

Appendix C.1. Per-Step Cost in the Surface Experiments

• On surface experiments, baseline updates cost 2.3–3.7 ms per step, Herrera methods add roughly −2% to 55%, memory methods add 40–105% on standard surfaces, and adaptive-memory methods cost about 2–3 times baselines.Adaptive-order methods fall between Herrera and memory groups, while related-work methods carry comparable machinery.Source: Appendix C.1 · p. 108, Table C.26 · p. 109
• The additive fractal variant adds approximately 5.1–5.8 ms to every step, reducing FCAdam_GL’s relative cost from about 3.2 times SGD to about 1.5 times.The perturbation ladder and gradient dominate increasingly as objective evaluation becomes more expensive.Source: Appendix C.1 · p. 108
• Ackley and Himmelblau have nearly identical per-step times within each variant, with differences of only fractions of a millisecond across optimizers.This indicates that measured differences primarily reflect update rules and perturbation costs rather than the specific surface.Source: Appendix C.1 · p. 108

Appendix C.2. Training Time in the Classification Experiments

• The Herrera group combines the highest cross-dataset accuracy with the second-lowest mean training time, averaging 6.5 s, while FAdam and FAdadelta average 5.5 s and 6.0 s.Adaptive-memory methods average 9.9 s, reflecting additional per-step machinery, with roughly 20% more training time than the standard group.Source: Appendix C.2 · p. 109
• Classification training-time rows mix optimizer cost with activation cost because each optimizer uses its best per-dataset configuration, producing apparent inversions relative to per-step measurements.GL-Memory averages 5.3 s despite higher per-step optimizer cost, whereas RMSprop reaches 9.7 s when its winning configurations use expensive fractal activations.Source: Appendix C.2 · pp. 109–110, Table C.27 · p. 110

Appendix C.3. Summary

• Fractional overhead is moderate on cheap objectives but shrinks with expensive evaluations, and neural-network training time varies less than per-step factors suggest.Herrera scaling typically adds around 10% and up to 55%, safeguarded Grünwald–Letnikov memory adds 40–105%, and adaptive-memory or related-work methods cost roughly 2–3 times baselines.Source: Appendix C.1 · p. 108, Appendix C.3 · p. 110
• Classification runtimes are governed at least as much by winning activation choices as by optimizers, so method-family comparisons must be read with accuracy results.Fractal activations use 30–100-term truncated series; the most expensive mean best-configuration training time was approximately 12 s.Source: Appendix C.2 · pp. 109–110, Appendix C.3 · p. 111
• The complete programme of 88 800 training runs and 2 × 1680 surface runs was feasible on standard hardware.Absolute runtimes remained inexpensive for the network sizes studied.Source: Appendix C.3 · p. 111