PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning using Rectified Linear Units (ReLU)

Abien Fred Agarap

arXiv:1803.08375v3cs.NEcs.CVcs.LGstat.ML

TL;DR

The paper addresses the mistaken attribution of ReLU’s origin to the 2018 study, which examined ReLU at the classification layer rather than introducing it as a hidden-layer activation. The paper compares ReLU, Tanh, and Sigmoid across image classification, text classification, and image reconstruction, and traces ReLU’s intellectual lineage. Across tasks, ReLU and Tanh consistently outperformed Sigmoid; ReLU reached 83.33% mean text accuracy, while Tanh led reconstruction at 33.19 dB on MNIST and 29.64 dB on CIFAR10 PSNR.

  • Problem

    The paper addresses the mistaken attribution of ReLU’s origin to the 2018 study, which examined ReLU at the classification layer rather than introducing it as a hidden-layer activation.

    Source: §1 · p. 2

  • Method

    The paper compares ReLU, Tanh, and Sigmoid across image classification, text classification, and image reconstruction, and traces ReLU’s intellectual lineage.

    Source: §1 · p. 2, §4 · p. 6

  • Results

    Across tasks, ReLU and Tanh consistently outperformed Sigmoid; ReLU reached 83.33% mean text accuracy, while Tanh led reconstruction at 33.19 dB on MNIST and 29.64 dB on CIFAR10 PSNR.

    Source: §3.2 · p. 5, §3.3 · p. 5, §4 · p. 6

  • Takeaways & Limitations

    The paper attributes ReLU’s integration into deep learning to Nair and Hinton (2010), not the 2018 study that used it at the classification layer.

    Source: §1 · p. 2, §4 · p. 6

    In the 1D CNN, all three activation functions’ validation losses reached a global minimum around epoch 3 before diverging, suggesting early stopping could improve performance.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

The Rectified Linear Unit (ReLU) is a foundational activation function in artficial neural networks. Recent literature frequently misattributes its origin to the 2018 (initial) version of this paper, which exclusively investigated ReLU at the classification layer. This paper formally corrects the citation record by tracing the mathematical lineage of piecewise linear functions from early biological models to their definitive integration into deep learning by Nair & Hinton (2010). Alongside this historical rectification, we present a comprehensive empirical comparison of the ReLU, Hyperbolic Tangent (Tanh), and Logistic (Sigmoid) activation functions across image classification, text classification, and image reconstruction tasks. To ensure statistical robustness, we evaluated these functions using 10 independent randomized trials and assessed significance using the non-parametric Kruskal-Wallis $H$ test. The empirical data validates the theoretical limitations of saturating functions. Sigmoid failed to converge in deep convolutional vision tasks due to the vanishing gradient problem, thus yielding accuracies equivalent to random probability. Conversely, ReLU and Tanh exhibited stable convergence. ReLU achieved the highest mean accuracy and F1-score on image classification and text classification tasks, while Tanh yielded the highest peak signal to noise ratio in image reconstruction. Ultimately, this study confirms a statistically significant performance variance among activations, thus reaffirming the necessity of non-saturating functions in deep architectures, and restores proper historical attribution to prior literature.

1 Introduction and Related Works

The introduction links earlier training difficulties to small activation ranges and network depth, then focuses on activation functions as one proposed way to address them. It also corrects ReLU attribution, distinguishing earlier foundations from its integration into deep learning by.

1 Introduction and Related Works

  • Small activation ranges and network depth contributed to gradients shrinking toward zero and networks stopping learning.The introduction also identifies insufficient computing power and training data, and sensitivity to initial weights, as historical difficulties.Source: §1 · p. 1
  • The paper compares ReLU and its variants with logistic and tangent activations, among proposed approaches to avoid the vanishing-gradient problem.The problem is described as gradients becoming too small during backpropagation for learning to continue.Source: §1 · pp. 1–2
  • The paper distinguishes early mathematical and biological foundations from ReLU’s integration into modern deep learning by.It notes that later proved ReLU prevents vanishing gradients in deep feedforward networks.Source: §1 · p. 2
  • The 2018 study used ReLU at the classification layer instead of Softmax; it did not introduce ReLU as a hidden-layer activation.The paper says the citation record should return attribution to.Source: §1 · p. 2

2 Methods

The methods compare three activation functions across image and text classification and image reconstruction, using task-specific neural architectures. They report datasets, preprocessing, experimental configurations, and evaluation measures for the comparison.

2.1 Hardware and Software Setup

  • The experiments used an Intel Core Ultra 7 265, 48GB DDR5 RAM, an Nvidia GeForce GTX 5060 Ti, PyTorch, and seed 42.Implementations were available at GitLab.Source: §2.1 · p. 2

2.2 Datasets and Preprocessing

  • Image classification used MNIST, FashionMNIST, CIFAR10, and CIFAR100; text classification used IMDB, while reconstruction used MNIST and CIFAR10.Images were resized to 32 × 32 pixels; text was tokenized with a pretrained BERT tokenizer and a 256-token maximum sequence length.Source: §2.2 · p. 2

2.3 Model Architectures

  • Image classification used a VGG-style CNN, text classification a 1D CNN, and reconstruction a convolutional autoencoder; each used ReLU, Tanh, or Sigmoid in hidden layers.Figure 1 depicts the image CNN, text classifier, and autoencoder architectures.Source: §2.3 · p. 2, Figure 1 · p. 3

2.4 Experimental Setup and Evaluation Metrics

  • Ten independent runs used distinct seeds; each dataset allocated 50% to training and 5% to validation, discarding the remaining 45% of training data.A withheld test set was reserved for final evaluation; significance was assessed with the Kruskal-Wallis H test.Source: §2.4 · pp. 2–3

3 Results

Across the three tasks, activation performance differed: Sigmoid failed to converge in image classification, ReLU led text classification, and Tanh produced the strongest image reconstructions. The discussion attributes deep-network learning differences to saturation and gradient flow.

3.1 Image Classification

  • Sigmoid failed to converge on the four image-classification datasets, ending near random-probability accuracy: roughly 10% on 10-class datasets and 1% on CIFAR100.The loss curves show the failure across MNIST, FashionMNIST, CIFAR10, and CIFAR100.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 Text Classification

  • 83.33% was ReLU’s highest mean accuracy on IMDB after 15 epochs, with a statistically significant performance advantage despite shared overfitting.The Kruskal-Wallis H test confirmed significant variance (p < 0.05).Source: §3.2 · p. 5

3.3 Image Reconstruction

  • 33.19 dB on MNIST and 29.64 dB on CIFAR10 were Tanh’s highest PSNR values, with ReLU following closely.Higher PSNR indicates superior image fidelity.Source: §3.3 · p. 5, Table 3 · p. 5

3.4 Discussion

  • Sigmoid saturates and rapidly vanishes gradients, while Tanh’s zero-centered outputs mitigate saturation slightly and ReLU avoids positive saturation.The paper says ReLU’s non-saturating property allows gradients to flow efficiently through deep architectures.Source: §3.4 · p. 6

4 Conclusion

The empirical comparison finds ReLU and Tanh outperform Sigmoid across image classification, text classification, and image reconstruction, with severe gradient vanishing in convolutional architectures. The paper also corrects ReLU’s historical attribution, identifying Nair and Hinton (2010) as its integration into deep learning.

4 Conclusion

  • ReLU and Tanh consistently outperform Sigmoid across image classification, text classification, and image reconstruction tasks.Sigmoid suffers from severe gradient vanishing in convolutional architectures.Source: §4 · p. 6
  • Sigmoid suffers from severe gradient vanishing in convolutional architectures.Source: §4 · p. 6
  • Nair and Hinton (2010) integrated ReLU into deep learning; the paper corrects misattribution to the 2018 study, which examined ReLU only at the classification layer.Source: §4 · p. 6

A Supplemental Results

Supplemental classification results report stable ReLU and Tanh metrics across 10 image-classification trials, while Sigmoid’s F1 scores are near zero. For text classification, ReLU has the highest F1 score, though Sigmoid exceeds Tanh across the reported metrics.

A Supplemental Results

  • Tanh marginally outperforms ReLU on all three extended metrics on CIFAR100, while ReLU and Tanh show low standard deviations, frequently below 0.01, across image-classification tasks.Sigmoid’s F1 scores are near zero with zero variance; the passage relates its predictions to random guessing, including 10% recall on 10-class datasets and 1% on the 100-class dataset.Source: A Supplemental Results · p. 9, Table 4 · p. 9
  • The supplemental tables cover mean precision, recall, and F1 with standard deviations across 10 randomized trials for four image-classification datasets and the IMDB sentiment dataset.Source: Table 4 · p. 9, Table 5 · p. 9
  • ReLU achieves the highest F1 score (0.8333) in text classification, while Sigmoid outperforms Tanh in precision, recall, and F1 score in this context.The reported ReLU standard deviation example is ±0.0039.Source: A Supplemental Results · p. 9, Table 5 · p. 9

Related papers

Loading 1803.08375v3…