Source-linked AI summary
HAND: A Biologically-Inspired Activation Function that Improves Generalisation and Sample Efficiency in Image Classification
Michael W. Spratling, Heiko H. Schütt
TL;DR
DNNs require substantial training data and time, while inductive bias may guide learning toward useful solutions with less data. This paper introduces HAND, a biologically inspired CNN activation combining homeostasis, an accelerating nonlinearity, and divisive-normalisation. HAND reaches the same ImageNet1k clean accuracy in 25 epochs that an unmodified ConvNeXt-tiny reaches in 200, with reported benefits especially when training or data are limited.
Problem
DNNs can learn mappings without strong built-in inductive biases given sufficient samples and training time, but such biases may guide learning toward useful solutions more efficiently.
Source: §1 · p. 1Method
HAND replaces standard CNN activations with a biologically inspired combination of BatchNorm homeostasis, a squared-ReLU nonlinearity, and divisive-normalisation.
Source: §1 · p. 2, §3.1 · p. 3Results
25 epochs of HAND-based ConvNeXt-tiny training matched the clean accuracy of the unmodified model after 200 epochs, and HAND typically improved performance most under limited training or data conditions.
Source: §1 · p. 2, §4.1 · p. 5, §5 · p. 9Takeaways & Limitations
Competition implemented through divisive-normalisation is an appropriate inductive bias for image classification, with HAND producing large improvements especially under limited epochs, augmentation, or class samples.
Source: §5 · p. 9Whether HAND improves other vision applications remains unverified, and architectures designed to exploit competition might perform better.
Source: §5 · p. 9
Abstract
from arXiv · showhide
DNNs exhibit robustness and generalisation issues not seen in humans. They are also far less data-efficient learners, requiring considerably more training samples to accurately classify novel exemplars. Inductive bias could help with these issues by providing in-built mechanisms to improve generalisation, and hence, reduce reliance on learning from data. We incorporate a biologically-inspired inductive bias into a new activation function, HAND (Homeostasis, Accelerating Nonlinearity, and Divisive-nomalisation), and show its effectiveness with CNNs trained on image classification. Using HAND a ConvNeXt-tiny required 25 training epochs to reach the same accuracy on ImageNet1k as the unmodified model achieved after 200 epochs. Consistent with the effects of an inductive bias, the performance gap reduced with training time and increased data augmentation. When the volume of training data was reduced and unevenly distributed between classes (Long-tailed ImageNet) the improvements in accuracy were even larger and did not reduce with increased training time. Generalisation performance with the common-corruptions data, and the ability to reject samples from unknown classes, were unaffected or improved by HAND. Results generalised across CNN architectures and training data-sets. HAND can, therefore, reduce the required training time and/or the required volume and variety of training data, helping to improve sample efficiency.
1 Introduction
The introduction argues that inductive bias can guide DNN learning toward useful solutions with less training time and fewer samples, despite the field’s emphasis on scaling. It proposes biologically inspired competition in CNNs and reports that HAND improves performance particularly when training resources or data are limited.
Motivation
- Inductive bias can narrow the candidate solutions and guide learning toward a good solution with less time and fewer training samples.The introduction links this possibility to frugal AI.Source: §1 · p. 1
- Prior biologically inspired DNN front-ends modify an early stage, and the introduction argues their effects are limited and potentially reproducible through data augmentation.The authors say more effective inductive biases remain to be proposed.Source: §1 · p. 1
- Competition is proposed because a single image location can contain only one feature, motivating competition among neurons representing alternative interpretations of the same patch.The authors hypothesize that competition could improve pattern discrimination and reduce redundant neuron representations.Source: §1 · p. 1
- Competition could reduce redundancy during learning, avoiding the need to train a large redundant model before post-hoc pruning.The introduction notes that pruning methods remove redundant neurons or connections from pretrained models.Source: §1 · pp. 1–2
Contribution
- HAND uses divisive-normalisation to create competition across channels at the same receptive-field location, and typically improves clean accuracy, corruption accuracy, and OOD rejection.The largest improvements occur with few training epochs, limited training-time augmentation, or reduced and imbalanced training data.Source: §1 · pp. 1–2
2 Related Work
The related-work section reviews normalization and competition mechanisms in neural networks. It distinguishes HAND’s across-channel pooling and accelerating nonlinearity from prior divisive-normalisation implementations and other competition approaches.
2 Related Work
- Prior neuron-response mechanisms can depend on the previous or current activity of other neurons; the review focuses on the most prominent or relevant examples.This section frames the scope of the related-work discussion.Source: §2 · p. 2
2.1 Normalisation Methods: Subtraction and Division
- Normalization layers standardize activation means and standard deviations to make DNN training faster and more stable.BatchNorm pools statistics over batch and spatial dimensions per channel, whereas LayerNorm, InstanceNorm, and GroupNorm use different pools.Source: §2.1 · p. 2
2.2 Divisive-normalisation Methods: Division Only
- Prior divisive-normalisation methods divide activity by a powered, weighted sum over a neighborhood, with implementations differing in whether the pool spans channels, space, or both.Some use local channel-and-space regions or spatial neighborhoods, while others pool across selected channels.Source: §2.2 · pp. 2–3
- HAND sums across all channels and uses an accelerating nonlinearity with α = 2, unlike most earlier implementations that used α = 1.The passage notes neuroscientific evidence supporting an α value of approximately two.Source: §2.2 · p. 3
2.3 Other Mechanisms of Competition
- Other competition mechanisms include k-WTA activations and weight-overlap inhibition, with reported effects on robustness, shape bias, or generalisation.The cited k-WTA adversarial-robustness improvement was modest and attributed to destroyed gradient information.Source: §2.3 · p. 3
3 Methods
HAND combines homeostasis, an accelerating nonlinearity, and divisive-normalisation, then replaces standard CNN activations under architecture-specific adjustments. The experiments specify training settings and evaluate classification, corruption generalisation, OOD rejection, and related benchmarks across datasets.
3.1 HAND: Homeostasis, Accelerating Nonlinearity, and Divisive-nomalisation
- HAND pools across all channels at a location and uses a single learned weight and semi-saturation constant per layer, reducing the activation’s learned parameters to two per layer.The weight starts at 1 divided by the number of channels, while the semi-saturation constant starts at one; learning typically strengthens normalization by increasing the weight.Source: §3.1 · p. 3
- HAND squares ReLU outputs after BatchNorm and applies divisive-normalisation; the nonlinearity separates active from inactive neurons, while normalization can stabilize learning and encourage distinct pattern responses.BatchNorm is interpreted as homeostasis that keeps neurons within stable operating conditions; the competition-related normalization gradients can favor weaker outputs from other neurons at the same location.Source: §3.1 · pp. 3–4
- The experiments replace activations differently by architecture: GELU throughout ConvNeXt blocks, ReLU in the first two ResNet block activations, and convolution-following ReLUs in AlexNet.ConvNeXt and ResNet also receive specified changes to preceding normalization or bias terms to avoid instabilities.Source: §3.1 · p. 4
3.2 Training Settings
- Training uses ImageNet1k and long-tailed ImageNetLT, whose class counts range from 5 to 1280, with architecture-specific standard recipes and basic or strong augmentation schemes.Strong augmentation adds TrivialAug, RandErase, mixup, cutmix, and label smoothing to random resized crops and horizontal flips.Source: §3.2 · p. 4
- Additional ConvNeXt-tiny experiments use SVHN, CIFAR10, CIFAR100, TinyImageNet, and long-tailed CIFAR variants with class imbalance ratios of 100 or 10.Long-tailed samples are removed unevenly across classes, and the ConvNeXt stem is adapted for small images.Source: §3.2 · p. 4
- Some long-tailed experiments use logit-adjusted loss, which modifies logits according to class sample proportions; selected runs that failed under default settings completed after reducing the initial learning rate tenfold.The failures included collapse or no validation improvement in specified experiments.Source: §3.2 · pp. 4–5
3.3 Performance Metrics
- Evaluation measures balanced ImageNet1k clean accuracy, average ImageNet-C corruption accuracy, and OOD rejection AUROC averaged across seven unknown-class datasets.OOD confidence uses Maximum Logit Score, the maximum network output before an activation function.Source: §3.3 · p. 5
- Additional evaluation uses ModelvsHuman accuracy and error consistency, corruption datasets for alternative training datasets, and three-repeat averages with two-sample t-tests for significance.ModelvsHuman spans 17 datasets; significance is assessed at p < 0.05.Source: §3.3 · p. 5
4 Results
On ImageNet1k, HAND let ConvNeXt-tiny reach in 25 epochs the clean accuracy the unmodified network required 200 epochs to attain. Improvements extended to scarce, imbalanced data and generally across CNNs and datasets, with effects qualified by augmentation and architecture.
4.1 Effects of Training Duration and Data Augmentation
- Stronger augmentation improved clean and corrupt accuracy but reduced OOD rejection for both activations, and eliminated HAND’s advantage by 200 epochs.With longer training, the performance gap narrowed; the clean-accuracy difference remained significant at 200 epochs under basic augmentation.Source: §4.1 · pp. 5–6
- At 25 epochs, HAND reached 73.76% clean accuracy versus GELU’s 71.24%; GELU required 200 epochs to match that accuracy.HAND’s 25-epoch result also had 36.77% corrupt accuracy and 91.21% OOD rejection AUROC.Source: §4.1 · p. 5, Figure 1 · p. 6
4.2 Effects of Data Scarcity and Imbalance
- On ImageNetLT, 25-epoch HAND clean and corrupt accuracy remained unmatched by GELU after 200 epochs, and HAND exceeded GELU in OOD rejection after 50 epochs.Logit-adjusted loss boosted standard ConvNeXt-tiny’s clean and corrupt accuracy, but HAND’s advantage remained and its performance was further boosted by that loss.Source: §4.2 · p. 6, Figure 2 · p. 7
4.3 Generalisation to Other CNN Architectures and Data-sets
- Across CNN architectures and datasets, HAND generally improved performance, but ResNet50 on ImageNetLT had mixed results and HAND did not enhance the vision transformer.On harder balanced datasets and additional long-tailed datasets, HAND improved performance across metrics; SVHN was mixed after performance had plateaued before 25 epochs.Source: §4.3 · pp. 6–7, Table 1 · p. 8
4.4 Ablation Study
- Ablations indicate the accelerating nonlinearity contributes most to HAND’s gains, while BatchNorm or divisive-normalisation can be needed for stability in some architectures.Removing squaring reduced performance on both architectures; divisive-normalisation was essential for stable AlexNet learning but its ConvNeXt-tiny performance reduction was not statistically significant.Source: §4.4 · p. 7, Table 2 · p. 8
4.5 Comparison with Previous Divisive-Normalisation Implementations
- HAND with larger AlexNet attained 59.60% clean accuracy versus Miller et al.’s 59.61%; prior divisive-normalisation methods had not been shown to scale to modern architectures or rigorously evaluated for generalisation and OOD rejection.Pan et al. reported approximately 64% using a larger AlexNet with two additional fully connected layers; the authors did not test HAND with that bespoke architecture.Source: §4.5 · p. 9
5 Discussion
The paper presents competition between neurons representing the same image patch as an appropriate inductive bias for image classification, with HAND improving performance especially under limited training or data. Its implementation depends on accelerating nonlinearity and homeostasis, while the authors note limits to the bias’s generality and leave broader applications for future work.
5 Discussion
- HAND improves image-classification performance especially with few training epochs, limited training-time augmentation, or few samples for some classes.The authors identify competition between neurons representing the same image patch, implemented using divisive-normalisation, as an appropriate inductive bias.Source: §5 · p. 9
- The implementation’s success relies on accelerating nonlinearity and homeostasis before divisive normalisation, with squaring and BatchNorm more important for performance than normalisation.The authors note that these components are also supported by neuroscientific data.Source: §5 · p. 9
- Increasing inductive bias may help address computational costs, data-curation constraints, and risks of learned prejudice, though the authors frame these as broader concerns.The passage also notes claims that AI may be running out of training data.Source: §5 · p. 9
- The bias may be less general when task-specific constraints are inappropriate; HAND’s potential beyond vision classification remains unverified, and existing CNNs may not fully exploit competition.The authors also suggest stronger competition might further improve rejection of unknown-class samples.Source: §5 · p. 9