Source-linked AI summary
Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks
Chunjie Luo, Jianfeng Zhan, Lei Wang, Qiang Yang
TL;DR
Unbounded dot-product pre-activations can increase neuron variance, harming generalization and slowing training. The paper replaces dot products with cosine or centered cosine similarity, then compares the resulting normalization with batch, weight, and layer normalization across several datasets. Cosine normalization and centered cosine normalization significantly reduce classification test error relative to those alternatives.
Problem
Unbounded dot products increase the risk of neuron variance, which can make models sensitive to input-distribution changes and aggravate internal covariate shift.
Method
The paper uses cosine similarity or centered cosine similarity, also called Pearson Correlation Coefficient, instead of dot product in neural networks.
Results
Cosine normalization and centered cosine normalization significantly reduce classification test error compared with batch, weight, and layer normalization.
Takeaways & Limitations
Cosine normalization provides a normalization approach that is robust to different input magnitudes and does not depend on batch or mini-batch statistics.
Takeaways & Limitations
Cosine similarity requires non-zero weight and input vectors because its denominator cannot be zero.
Abstract
from arXiv · showhide
Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change of input distribution, thus results in poor generalization, and aggravates the internal covariate shift which slows down the training. To bound dot product and decrease the variance, we propose to use cosine similarity or centered cosine similarity (Pearson Correlation Coefficient) instead of dot product in neural networks, which we call cosine normalization. We compare cosine normalization with batch, weight and layer normalization in fully-connected neural networks as well as convolutional networks on the data sets of MNIST, 20NEWS GROUP, CIFAR-10/100 and SVHN. Experiments show that cosine normalization achieves better performance than other normalization techniques.
1. Background and Motivation
The paper motivates replacing unbounded dot-product pre-activations because large neuron variance can harm generalization and slow training. Cosine similarity is proposed as a way to bound this quantity without relying on batch statistics.
- Large neuron variance makes models sensitive to input-distribution changes and can worsen generalization.
- Variance can be amplified across layers, especially in deep networks, aggravating internal covariate shift.
- Dot-product pre-activations are unbounded, increasing the risk of large neuron variance.
- Weight decay and max normalization reduce weights to alleviate the problem, while batch normalization uses mini-batch statistics.
- The paper combines cosine similarity with neural networks to thoroughly bound the dot product.
2. Cosine Normalization
Cosine normalization replaces dot product with cosine similarity in the pre-activation, bounding it between -1 and 1 and narrowing neuron variance. The method supports backpropagation without batch-statistics dependence, with implementation caveats for activation functions, zero vectors, and convolutional receptive fields.
- Cosine normalization uses cosine similarity instead of dot product when computing a neural-network pre-activation.
- In a simple network, the hidden-unit output applies the nonlinear function to the cosine-normalized pre-activation.
- The normalized pre-activation is bounded between -1 and 1, controlling neuron variance within a narrow range.
- ReLU requires no re-scaling or re-shifting empirically, whereas sigmoid, tanh, and softmax require re-valuing the normalized result.
- Fully connected networks divide by weight and input norms, while convolutional networks normalize inputs within receptive fields with different norms.
- Cosine normalization requires non-zero vectors because the denominator cannot be zero; non-zero bias can avoid zero vectors.
- Cosine-normalized networks use backpropagation, support batch or stochastic gradient descent, and compute identically during training and inference.
3. Discussions
The discussions contrast cosine normalization with weight and layer normalization and connect it to similarity-based learning. Cosine normalization additionally constrains inputs and weights, improving robustness to input magnitude and differing from layer normalization in convolutional settings.
- Weight normalization constrains weights, whereas cosine normalization additionally divides by input magnitude to narrow pre-activation variance.
- Scaling an input from [1, 2] to [10, 20] changes softmax output from [0.2689, 0.7311] to [0, 1] without cosine normalization.
- Cosine normalization can keep output distributions stable when input magnitude varies because it depends on the angle between input and weight.
- Input scaling changes gradient scaling in weight normalization, while cosine normalization offsets the factor through the input norm denominator.
- Pearson Correlation Coefficient constrains both weights and inputs, while layer normalization constrains only inputs.
- In convolutional networks, Pearson Correlation Coefficient uses receptive-field statistics, whereas layer normalization uses whole-layer statistics.
- The paper presents cosine similarity and Pearson Correlation Coefficient as basic metrics for end-to-end deep networks trained by backpropagation.
4. Experiments
The experiments compare cosine, centered cosine, batch, weight, and layer normalization across fully connected and convolutional networks. Results are reported against normalized baselines because unnormalized networks performed much worse.
- The study evaluates normalization techniques on fully connected networks using MNIST and 20NEWS GROUP.
- Convolutional networks are evaluated on CIFAR-10, CIFAR-100, and SVHN with multiple normalization techniques.
- Unnormalized fully connected and convolutional networks perform much worse than networks using normalization.
4.1. Date sets
The experiments use MNIST, 20NEWS GROUP, CIFAR-10/100, and SVHN, covering handwritten digits, topic classification, natural images, and house-number recognition.
- MNIST: MNIST contains 28x28 grayscale handwritten-digit images for 10-class classification, with 60,000 training and 10,000 test examples.Pixel values are scaled to [0, 1] before modeling.
- 20NEWS GROUP: 20NEWS GROUP contains 20-topic document classification data, reduced to 11,200 training and 7,500 test examples after random dropping.Words with document frequency greater than 5 produce 21,567 feature dimensions.
- CIFAR-10/100: CIFAR-10 and CIFAR-100 contain natural 32x32 RGB images with 10 and 100 classes, respectively.CIFAR-10 has 50,000 training and 10,000 test images; augmentation includes cropping, flips, brightness distortion, and contrast distortion.
- SVHN: SVHN contains 604,388 training-plus-extra images and 26,032 test images for centered-digit classification in 32x32 RGB images.The data are augmented using the CIFAR-10/100 procedure.
4.2. Protocols
The study evaluates fully connected networks on MNIST and 20NEWS GROUP and a VGG-like convolutional architecture on CIFAR-10/100 and SVHN using mini-batch gradient descent.
- Fully connected networks: MNIST and 20NEWS GROUP use two-hidden-layer fully connected networks, with 1000 units per hidden layer and task-specific softmax outputs.The final layer has 10 classes for MNIST and 20 classes for 20NEWS GROUP.
- Convolutional networks: CIFAR-10/100 and SVHN use a VGG-like architecture with 12 weighted layers.The architecture includes repeated convolutional and max-pooling blocks followed by fully connected layers.
- Convolutional networks: Each convolutional layer uses a 3×3 receptive field with stride 1, while each max-pool layer uses 2×2 regions with stride 1.These settings define the spatial operations in the convolutional architecture.
- Training protocol: ReLU activations are used in hidden layers, weights are initialized from a truncated normal distribution with 0 mean and 0.1 variance, and mini-batch gradient descent trains the networks.Batch sizes are 100 for fully connected networks and 128 for convolutional networks; hidden layers use no rescaling or reshifting after normalization.
4.3. Results
Across MNIST, 20NEWS GROUP, CIFAR-10/100, and SVHN, centered cosine normalization generally gives the lowest test error, while cosine normalization is consistently competitive. The results also report similar convergence speed and greater test-error variance for batch normalization in several settings.
- MNIST: 1.39% is the lowest MNIST mean test error, achieved by centered cosine normalization over the last 50 epochs.Cosine and layer normalization achieve 1.40% and 1.43%, respectively; batch normalization fluctuates because mini-batch statistics change.
- Training behavior: Convergence speeds are close across normalization techniques, while cosine normalization accelerates training as well as other normalization methods.On MNIST, centered cosine and cosine normalization achieve similar test errors and are slightly better than layer normalization.
- 20NEWS GROUP: 29.37% and 31.73% are the lowest and second-lowest 20NEWS GROUP test errors, achieved by centered cosine and cosine normalization, respectively.Batch normalization performs worst at 43.94%, while batch and weight normalization also have larger test-error variance than other methods.
- CIFAR-10: 6.39% and 7.33% are the lowest and second-lowest CIFAR-10 test errors, achieved by centered cosine and cosine normalization, respectively.Layer normalization reaches 7.42%, batch normalization 8.08%, and weight normalization 8.55%; batch normalization has larger variance.
- CIFAR-100: 27.49% is the lowest CIFAR-100 test error, achieved by centered cosine normalization; cosine and batch normalization are nearly tied at 31.02% and 31.01%.Batch normalization has larger variance, and weight normalization has the highest test error at 37.87%.
- SVHN: 2.22% and 2.34% are the lowest and second-lowest SVHN test errors, achieved by centered cosine and cosine normalization, respectively.Batch, layer, and weight normalization achieve 2.49%, 2.58%, and 2.63%, respectively.
5. Conclusions
The paper proposes cosine normalization for neural networks, using cosine or centered cosine similarity instead of dot products and evaluating it across network types and datasets. Experiments show that cosine and centered cosine normalization significantly reduce classification test error compared with batch, weight, and layer normalization.
- 5. Conclusions: Cosine normalization replaces dot products with cosine similarity or centered cosine similarity, also called the Pearson correlation coefficient.It is evaluated in both fully connected and convolutional networks across MNIST, 20NEWS GROUP, CIFAR-10/100, and SVHN.
- 5. Conclusions: Cosine normalization bounds neuron pre-activations within a narrower range, lowers neuron variance, and improves robustness to different input magnitudes.The method does not depend on batch or mini-batch statistics and uses the same forward computation during training and inference.
- 5. Conclusions: Cosine and centered cosine normalization significantly reduce classification test error compared with batch, weight, and layer normalization.The comparison covers the network types and datasets described in the paper.