Source-linked AI summary
Gaussian Neural Networks
Peter Kuhn, Victoria Heusinger-Heß
TL;DR
Standard neural networks lack explicit priors over expected activations, while activation-space priors remain relatively unexplored. GaNNs model hidden-layer activities with learnable Gaussian priors and an additional unsupervised loss. They report better predictive performance across classification and regression tasks than standard architectures with standard regularizers, while uncertainty results are mixed.
Problem
Standard neural networks lack an explicit representation of expected activation patterns, and activation-space priors remain relatively unexplored.
Source: §1 · p. 1Method
GaNNs model hidden-layer activities as noisy signals and add an unsupervised loss to learn Gaussian activity priors alongside the supervised objective.
Source: §1 · pp. 1–2, §3.1 · pp. 4–5Results
GaNNs achieve better predictive performance across classification and regression tasks than standard architectures trained with standard regularizers, while uncertainty results are mixed.
Source: §1 · p. 2, §5.1 · p. 11, §5.2 · p. 13Takeaways & Limitations
The sparse noise model is reported to be more computationally and memory efficient than the dense model and to converge more stably.
Source: §6 · p. 13The study focuses on pure Gaussian models, leaving combinations with standard architectures and evaluation in deeper, convolutional, or attention models for further study.
Source: §6 · p. 14
Abstract
from arXiv · showhide
Gaussian neural networks (GaNNs) are proposed as a novel regularization mechanism for neural networks. From a Bayesian perspective standard regularization techniques can be viewed as imposing priors over weight-space. Assuming priors over activation-space remains a largely unexplored possibility. GaNNs assume such priors. They do this by treating activities from earlier layers like signals with Gaussian noise and predicting the properties of the noise distribution using an additional unsupervised loss. While training, the unsupervised loss acts as a penalty on unexpected activities, allowing greater weight updates in less surprising directions. The paper demonstrates the superiority of Gaussian neural networks over standard neural networks on a variety of classification and regression tasks. We also investigate the ability of GaNNs to quantify uncertainty.
1 Introduction
Standard neural networks lack an explicit representation of expected activation patterns, motivating a regularizer based on learnable priors over hidden-layer activities. GaNNs add an unsupervised objective for those activities and report stronger predictive performance across classification and regression tasks, with mixed uncertainty results.
1 Introduction
- Standard neural networks have no explicit representation of expected neuronal activation vectors, limiting how they distinguish expected activities from surprises.The paper also notes that standard neural networks are not well suited to quantify aleatoric or epistemic uncertainty.Source: §1 · p. 1
- GaNNs treat hidden-layer activities as noisy or uncertain and induce learnable priors in activation space.The paper describes this as a novel regularization technique.Source: §1 · p. 1
- An extended abstract for the paper was accepted for publication at ICONIP 2026.Source: §1 · p. 1
- GaNNs outperform standard neural architectures with standard regularizers across a variety of classification and regression tasks, while uncertainty-quantification results are mixed.The paper also reports low computational and memory costs, with as few as two additional parameters per neuron depending on the details.Source: §1 · p. 2
2 Related Work
Related work covers Bayesian neural networks and dropout as approaches to uncertainty, weight decay as a weight-space prior, and Gaussian variance heads for modeling data noise. These methods provide context for the paper’s activation-space priors.
2.1 Bayesian Approaches to Neural Networks
- Bayesian neural networks place Gaussian priors on weights, can quantify epistemic uncertainty, and are less prone to overfitting, but require more computation and triple the parameter count.Dropout approximates Bayesian neural network functioning without increasing model parameters; Monte-Carlo dropout can quantify predictive uncertainty [1].Source: §2.1 · p. 2
2.2 The Bayesian Perspective on Regularization
- From a Bayesian perspective, weight decay corresponds to an isotropic Gaussian prior on weights that assigns higher prior probability to lower weights.Weight decay is presented as an established strong default regularizer.Source: §2.2 · p. 3
2.3 Gaussian Variance Heads
- Gaussian variance heads predict data-point-specific variance and are trained with a Gaussian loss that learns both the output mean and variance.The loss is a generalization of mean squared error, equivalent to it when variance is uniformly 1; expected-error modeling can reflect both epistemic and aleatoric uncertainty [4,5,9].Source: §2.3 · pp. 3–4
3 Gaussian Neural Networks
GaNNs learn Gaussian priors over hidden-layer activities and penalize departures from expected activity regimes as an activation-space regularizer. The section describes sparse and dense prior models, noisy predictive sampling, and metrics for evaluating predictive uncertainty.
3.1 Inner Gaussian Loss
- GaNNs place learnable Gaussian priors on hidden-layer activities, and prior violations add a loss penalty that biases weight updates toward learned activity regimes.The input and output layers have no activity priors, and the likelihood assumes noise at each layer is independent of noise at previous layers.Source: §3.1 · pp. 4–5
- The inner loss learns activity priors without labels and is combined with a supervised base loss using a no-gradient operator and hyperparameter α.The authors caution that directly summing the losses or using fixed weights can let the Gaussian loss dominate, depending on dataset and loss-scale evolution.Source: §3.1 · pp. 5–6
3.2 Dense and Sparse Gaussian Layers
- Sparse variance priors depend on associated activities, whereas dense priors depend on all preceding nodes and can model more complex activity patterns at greater computational and memory cost.The sparse model adds three parameters per node, while the dense model's parameter count grows quadratically with the number of Gaussian nodes; non-standard parameters can be dropped for feed-forward inference after training.Source: §3.2 · pp. 6–8, Figure 1 · p. 7
Noisy Sampling.
- At inference, GaNNs can sample learned activity-noise distributions to produce predictive distributions, with hyperparameter γ controlling noise strength.The authors do not learn γ here and caution that predictive uncertainty is not necessarily calibrated by default.Source: Noisy Sampling · p. 8
Performance Metrics.
- Uncertainty evaluation uses predictive-distribution NLL for regression and classification, and entropy-based AUROC to assess whether uncertain predictions identify classification errors.The proposed metrics do not disentangle different kinds of uncertainty.Source: Performance Metrics · pp. 8–9
4 Experiments
Experiments compare dense and sparse GaNNs with otherwise similar ANNs across ten diverse datasets. They use repeated seeded training and a shared training setup with task-specific choices.
4.1 Model Implementation
- Dense GaNN layers double linear units and split activities from variances, whereas sparse layers use a custom layer; comparison ANNs otherwise match GaNN structure.Gaussian layers return internal state for the supervised loss; dropout attaches only to activity nodes in GaNNs.Source: §4.1 · pp. 9–10
4.2 Training Setup
- Training averages five random-seed runs with SGD, task-specific initial learning rates, and a tenfold learning-rate decay after 30 epochs.The base loss is MSE for regression and cross-entropy for classification; α is set to 0.1 for simple classification and 0.01 for CIFAR100 and Tiny ImageNet, with v=0.5 throughout.Source: §4.2 · p. 10
4.3 Datasets
- Evaluation spans ten visual and tabular datasets, split evenly between regression and classification, including airfoil, yacht, UTK face, wine, million songs, CIFAR100, Tiny ImageNet, MNIST, and Fashion-MNIST.Table 1 summarizes dataset properties and the number of epochs used.Source: §4.3 · pp. 10–11, Table 1 · p. 11
5 Results
Across ten datasets, sparse GaNNs outperform the basic ANN on nine and tie on one, while dense GaNNs outperform it on seven and tie on one. The results also describe delayed convergence, hyperparameter behavior, and uncertainty-quantification performance.
5.1 Overall Performance
- Dense GaNNs are not necessarily superior to sparse variants and may converge late on Airfoil and Yacht as an overpowerful noise model prioritizes inner-loss minimization.On Airfoil, optimizing the inner loss first and the base loss later accompanies delayed convergence and higher final performance.Source: §5.1 · pp. 11–12, Figure 3 · p. 13
- 9/10 datasets: sparse GaNNs outperform ANN, with one tie; dense GaNNs do so on seven, also with one tie.Across the experiments, GaNN performance is more consistent than weight decay or dropout.Source: §5.1 · p. 11, Table 2 · p. 12
- α values of 0.5 and above collapse performance; within a reasonable range, performance is insensitive to v, whose main function is numerical stability.The CIFAR100 ablation plots validation accuracy over different α values.Source: §5.1 · p. 12, Figure 2 · p. 12
5.2 Uncertainty Quantification
- Both GaNN variants outperform competitors on regression uncertainty quantification, while classification results are less consistent.The best overall performance is described as arguably coming from combining GaNNs with dropout.Source: Table 3 · p. 13, §5.2 · p. 13
6 Discussion
GaNNs had a consistent positive performance impact, and the sparse noise model outperformed the dense model in efficiency and convergence stability. The discussion notes that the work focused on pure Gaussian models and identifies hybrids, deeper networks, other architectures, and other distributions as directions for further study.
6 Discussion
- The sparse noise model proved superior to the dense model in computational and memory efficiency, with a tendency toward more stable convergence.Source: §6 · p. 13
- Generally better regression results might indicate that regression models fit the Gaussian assumptions better.Source: §6 · p. 14
- The paper focused on pure Gaussian models, while combining standard architectures with a few Gaussian layers might retain positive effects.Source: §6 · p. 14
- Further study could examine deeper models, interactions with convolutions and attention, and generalization beyond Gaussian distributions.Source: §6 · p. 14