PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Networks Fail to Learn Periodic Functions and How to Fix It

Liu Ziyin, Tilman Hartwig, Masahito Ueda

arXiv:2006.08195v2cs.LGstat.ML

TL;DR

The paper addresses the lack of satisfactory neural-network methods for learning periodic functions when extrapolation beyond observed data is required. It studies activation-function extrapolation and introduces Snake, x + sin^2(x), to provide a periodic inductive bias. Snake learns periodic behavior in toy regression while maintaining competitive performance in image classification and supporting real prediction tasks.

  • Problem

    Existing neural-network methods provide limited guidance for learning periodic functions, especially when the function must be extrapolated beyond observed data.

  • Method

    The paper studies neural-network extrapolation and proposes Snake, the activation function x + sin^2(x), with a frequency factor to control its periodic component.

  • Results

    Snake learns the correct frequency while interpolating and extrapolating a simple periodic function, and achieves similar performance to competitive baselines, including 94.1% accuracy on ResNet-101.

  • Takeaways & Limitations

    A periodic activation can combine periodic extrapolation with favorable optimization behavior and serve as a straightforward alternative to ReLU in supported tasks.

Abstract

from arXiv · show

Previous literature offers limited clues on how to learn a periodic function using modern neural networks. We start with a study of the extrapolation properties of neural networks; we prove and demonstrate experimentally that the standard activations functions, such as ReLU, tanh, sigmoid, along with their variants, all fail to learn to extrapolate simple periodic functions. We hypothesize that this is due to their lack of a "periodic" inductive bias. As a fix of this problem, we propose a new activation, namely, $x + \sin^2(x)$, which achieves the desired periodic inductive bias to learn a periodic function while maintaining a favorable optimization property of the ReLU-based activations. Experimentally, we apply the proposed method to temperature and financial data prediction.

1 Introduction

Periodic functions matter across natural, biological, societal, and financial systems, yet neural-network methods lack a satisfactory way to learn them when periodicity is not known in advance. This work studies that gap, showing standard activations are insufficient outside the bounded training region and proposing a solution tested on toy and real tasks.

  • Periodic dynamics appear in natural cycles, biological clocks, transportation demand, stock markets, and broader economic fluctuations.
  • When periodicity is known beforehand, Fourier-space or transformed approaches can solve the problem, but unknown periodicity requires a model flexible enough for periodic and non-periodic functions.
  • The paper studies neural-network extrapolation beyond bounded regions and shows standard activations cannot learn periodic functions outside the observed-data range.
  • The proposed solution is tested on toy examples and real tasks, while the design of better activation functions or methods remains open.

2 Inductive Bias and Extrapolation Properties of Activation Functions

Periodic-function learning depends on extrapolating patterns beyond the bounded region containing observations. The paper argues that standard activation functions impose incompatible asymptotic behaviors, preventing periodic extrapolation.

  • Periodic functions repeat beyond the observed interval, so learning them requires both bounded-region fitting and extrapolation of the learned pattern.
  • A one-hidden-layer network with 512 neurons is trained on four analytical functions sampled outside a gap to compare activation-function interpolation and extrapolation.
  • ReLU networks asymptotically converge to a direction-dependent linear transformation, whereas tanh networks extrapolate like a constant function.
  • These asymptotic behaviors prevent ReLU and tanh networks from extrapolating periodic functions, and the argument extends to continuous activations asymptotically approaching ReLU or tanh, including Swish and Leaky-ReLU.

3 Proposed Method: x + sin2(x)

The paper proposes Snake, the activation x + sin^2(x), to combine periodic inductive bias with favorable optimization behavior. Experiments and analysis indicate that Snake captures periodic structure while remaining easier to optimize than several periodic alternatives.

  • Proposed activation: Snake is defined as x + sin^2(x), with a frequency parameter a that controls the periodic component.Larger a produces higher frequency behavior.
  • Optimization motivation: Sin-based activations can be difficult to optimize because non-monotonicity creates infinitely many local minima.The difficulty arises because shifting preactivations by 2π gives the same function.
  • Optimization motivation: Snake and x + sin(x) are easier to optimize than other periodic baselines and, on MNIST, than standard ReLU.The paper attributes the optimization challenge of sin to its non-monotonicity and seeks monotonicity and periodicity together.
  • Approximation rationale: The paper hypothesizes that Snake has better approximation properties than x + sin(x) because its expansion contains a non-vanishing second-order term.By contrast, the first nonlinear term of x + sin(x) appears at third order.
  • Regression experiment: When regressing sin(x), Snake captures the underlying frequency in both interpolation and extrapolation regions, whereas ReLU and tanh do not capture the periodic pattern.Snake’s learned amplitude differs somewhat from the ground truth.

4 “Universal Extrapolation Theorem”

The paper proves a universal extrapolation theorem for sufficiently wide Snake networks approximating piecewise C1 periodic functions on the real line. This extends bounded-region approximation with periodic extrapolation, while the authors note that the best practical activation variant remains unresolved.

  • Universal Extrapolation Theorem: A sufficiently wide one-hidden-layer Snake network can approximate any piecewise C1 periodic function with period L as width tends to infinity.The theorem concerns approximation on an unbounded domain.
  • Universal Extrapolation Theorem: For all x ∈ R, the approximation converges point-wise, and convergence is uniform when the target function is continuous.These are the theorem’s stated convergence guarantees.
  • Corollary: The theorem implies the classical bounded-region approximation result for continuous functions.The corollary uses a sufficiently wide two-layer network on a bounded interval.
  • Implications: Snake combines approximation ability on bounded regions with the ability to learn periodicity on unbounded regions.The paper reports practical usefulness comparable to standard tasks and significant outperformance on periodic-function learning.
  • Scope and open question: The authors leave open whether better activation functions or methods exist and recommend larger-scale experiments to compare periodic variants.This includes Snake, sin, cos, x + sin(x), and x + cos(x).

5 Initialization for Snake

The section develops initialization and post-activation corrections for Snake to maintain suitable signal variance, with settings adapted to task periodicity.

  • Variance-preserving initialization: Snake initialization is chosen so each layer’s output has unit variance, avoiding forward-signal divergence or vanishing.The leading-order approximation treats Snake as an identity function when selecting the weight variance.
  • Variance-preserving initialization: Uniform initialization uses a range described as a factor of 2 smaller than Kaiming uniform initialization.
  • Higher-order correction: An exact variance solution is provided for the parameterized Snake activation when higher-order corrections are needed.The exact solution depends on a.
  • Post-activation correction: Dividing post-activation values by σ_a corrects an additional bias induced by the sin^2(x) term.The correction is motivated by the nonlinear term’s contribution to the response.
  • Task-dependent settings: The correction improves training speed and converged accuracy on ResNet-101 with CIFAR-10.For standard tasks, 0.2 ≤ a ≤ amax is reported to work well, with a default of 0.5; periodic tasks usually favor a from 5 to 50.

6 Applications

The paper evaluates Snake across image classification, temperature prediction, financial forecasting, and recurrent-network comparison. Snake is competitive on CIFAR-10 and improves periodic extrapolation in several real and synthetic tasks.

  • 6.1 Image Classification: Snake matches ReLU and Leaky-ReLU in CIFAR-10 learning speed and final performance, and reaches 94.1% accuracy on ResNet-101.The ResNet-18 comparison uses roughly 10M parameters, while ResNet-101 has four times more parameters.
  • 6.2 Atmospheric and Body Temperature Prediction: Snake successfully extrapolates atmospheric temperature with the correct period, while tanh, ReLU, Leaky-ReLU, and Swish baselines fail to optimize satisfactorily.The Snake model achieves vanishing training and generalization loss; the baseline models do not reach zero training loss.
  • 6.2 Atmospheric and Body Temperature Prediction: Snake captures the body-temperature oscillation, predicting a peak around 16pm and a minimum around 4am despite no data point at 4am.The average hourly prediction is computed over a 30-day period.
  • 6.3 Financial Data Prediction: Snake outperforms competitors in Wilshire 5000 market-value prediction and is the only method reported to predict a recession during and beyond the testing period.The authors hypothesize that Snake captures long-term economic cycles in the trend.
  • 6.4 Comparison with RNN on Regressing a Simple Periodic Function: On a noisy periodic time-series task, Snake on a feedforward network significantly outperforms a simple ReLU-activated RNN and requires less computation.The task uses y = sin(0.1x), training length T = 100, and validation from t = 101 to 300.

7 Conclusion

The conclusion identifies extrapolation as central to neural-network optimization and generalization. Standard activations have mismatched inductive bias for periodic functions, while the proposed approach addresses this limitation.

  • 7 Conclusion: Standard activation functions struggle to optimize and generalize periodic functions beyond the observed data range because their inductive bias is mismatched.The conclusion frames extrapolation properties as a key ingredient for understanding neural-network optimization and generalization.

Broader Impact Statement

The paper argues that studying neural-network extrapolation matters because many natural and societal processes are periodic. Accurate periodic modeling could therefore have broad practical importance.

  • Broader Impact Statement: Periodic-function modeling may affect many fields because periodic processes occur throughout nature and society.The statement connects this practical scope to the broader importance of understanding extrapolation beyond observed data.

A Additional Experiments

Additional experiments show that Snake's parameter a controls frequency bias and that larger values improve extrapolation on more complex periodic functions. The section also compares Snake with tanh, ReLU, and sinusoidal alternatives.

  • A Additional Experiments: Larger a biases Snake toward higher frequencies, while smaller a favors lower frequencies.This behavior is demonstrated across different kinds of periodic functions, including continuous, discontinuous, and compound periodicity.
  • A Additional Experiments: For rectangular-function regression, increasing a improves Snake's extrapolation.The figure compares models using different values of a.
  • A Additional Experiments: For sin(x) + sin(4x)/4, larger a improves extrapolation and can reveal the higher-frequency modulation as a second learned signal.The a = 1 model treats the modulation as noise, whereas the a = 16 model appears to learn it.
  • A Additional Experiments: The additional figures compare Snake, tanh, and ReLU for simple sine regression and EUR-USD exchange-rate prediction.These comparisons extend the activation-function evaluation beyond the main application experiments.
  • A Additional Experiments: A recurrent model is also tested on the periodic dynamical system x(t) = cos(t/2) + 2sin(t/3) using alternative activation functions.The training region is marked by dashed vertical lines in Figure 13.

A.3 Currency exchange rate modelling

Snake models the EUR/USD exchange rate on the training range and produces the most realistic extrapolation beyond 2015. Its broader evaluations show comparable CIFAR-10 performance to ReLU-style activations, while variance correction improves deep-network training.

  • A.3 Currency exchange rate modelling: Snake is the only tested activation that models the exchange rate on the training range and makes the most realistic prediction beyond 2015.The experiment uses a two-hidden-layer network with 256 and 64 hidden neurons, SGD, and a learnable Snake frequency parameter.
  • CIFAR-10 comparisons: 93.5 ± 1.0% accuracy is achieved by both Snake and ReLU on CIFAR-10, with similar generalization performance and optimization speed.The comparison uses ResNet18 across a range of Snake frequency values.
  • CIFAR-10 comparisons: Snake achieves similar performance to ReLU and Leaky-ReLU when evaluated on CIFAR-10 with ResNet101.This experiment tests whether the proposed activation scales to larger and deeper networks.
  • A.6.1 Effect of Variance Correction: Variance correction leads to better training speed and better converged accuracy on ResNet101 trained on CIFAR-10.The benefit is reported as most pronounced for deep networks.

B Proofs for Section 2.2

The proofs characterize the asymptotic behavior of ReLU and tanh feedforward networks. ReLU networks approach direction-dependent linear transformations, whereas tanh networks approach constants, preventing periodic extrapolation.

  • ReLU extrapolation: ReLU feedforward networks asymptotically converge to a linear transformation determined by the extrapolation direction.The proof proceeds by analyzing a two-layer network and iterating the argument across layers.
  • ReLU extrapolation: For ReLU networks, the first hidden layer eventually has a fixed activation pattern, reducing subsequent computation to an affine transformation.The proof expresses the active-unit pattern with a diagonal matrix and defines the resulting transformation and bias.
  • Tanh extrapolation: Tanh feedforward networks asymptotically approach a constant vector as inputs diverge along any fixed direction.Each first-layer preactivation tends toward positive or negative infinity, so tanh outputs converge to ±1 before later layers process the resulting constant vector.

C Universal Extrapolation Theorems

The universal extrapolation results show that Snake networks can approximate periodic functions, including through Fourier representations, while retaining ordinary approximation capability on bounded intervals.

  • Periodic approximation: A one-hidden-layer Snake network with width N can converge uniformly to any piecewise C1 periodic function as N approaches infinity.The theorem states point-wise convergence generally and uniform convergence when the target function is continuous.
  • Fourier representation: Cos-activation networks can converge point-wise to any piecewise C1 periodic function by representing Fourier series to arbitrary order.The proof invokes the Fourier convergence theorem and constructs the series order by order.
  • Fourier representation: A finite number of Snake neurons can represent a single cosine activation neuron.This establishes the connection between Snake networks and the cosine-based approximation construction.
  • Fourier representation: A Snake network with 4m hidden neurons can represent exactly a Fourier series through order m.The construction follows from representing cosine components with finite combinations of Snake neurons.
  • Bounded-interval approximation: For any bounded continuous function on [a,b], sufficiently wide two-layer networks can approximate the function within ϵ.The result follows by matching the bounded interval to the periodic-approximation theorem’s region.
Loading 2006.08195v2…