Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks

1. Introduction

• More than 1% at every batch size, FRN outperforms other approaches on ResNetV2-50 ImageNet classification.Figure 1 reports validation performance for models trained on 8 GPUs at different batch sizes.Source: Figure 1 · p. 1, §1 · p. 1
• Prior alternatives either fail to match BN at large batch sizes, degrade at smaller sizes, or impose architectural constraints such as GN’s channel-group requirement.The paper presents FRN as avoiding these shortcomings through batch-independent operation.Source: §1 · p. 1
• BN’s reliance on large batches creates memory demands that trade off model capacity against processing larger inputs for detection, segmentation, and video.The paper says FRN maintains consistent performance across a range of batch sizes.Source: §1 · pp. 1–2
• FRN combines per-sample, per-filter normalization without mean subtraction and a learned-threshold TLU activation.The paper also reports advantages over BN and GN on ImageNet and over other methods on COCO object detection.Source: §1 · p. 2

2. Related work

• BN normalizes each activation channel using mini-batch mean and variance, but degrades with smaller mini-batches [31].The related-work discussion groups alternatives into methods reducing train-test discrepancy and methods avoiding batch normalization.Source: §2 · p. 2
• LN uses all layer channels, IN uses each sample’s channel responses, and GN normalizes channel groups; GN performs worse than BN at large batches and constrains channel counts [2] [35].GN avoids small-batch degradation, but its group size is experimentally determined and normalized layers must have channel counts that are multiples of that size.Source: §2 · pp. 2–3
• Other approaches include Weight Normalization’s filter reparameterization, Normalization Propagation’s idealized moment estimates, and divisive normalization over neighboring activations [28] [1].The discussion also situates divisive normalization in several other research contexts.Source: §2 · p. 3

3.1. Filter Response Normalization with Thresholded Activation

• FRN normalizes each channel over its spatial extent without mean subtraction, removing scaling from filter weights and pre-activations.The method uses a per-channel basis rather than normalizing across channels.Source: §3.1 · p. 3

3.1.1 Thresholded Linear Unit (TLU)

• TLU applies a learned threshold around ReLU, equivalent to a shared bias before and after ReLU, and significantly improves FRN performance.The paper hypothesizes that this form is more favorable for optimization.Source: §3.1.1 · p. 3

3.2. Gradients of FRN Layer

• With ϵ = 0, gradients with respect to x are orthogonal to x; gradients suppressed by ReLU can instead update τ.The orthogonality follows from a projection that removes the component in the direction of normalized x; the paper notes this property is known to benefit optimization.Source: §3.2 · p. 4

3.3. Learning ϵ

• For N=1, very small ϵ makes FRN step-like with very small gradients, so fully connected or low-dimensional models use learned ϵ initialized at 10^-4.Other models use a fixed ϵ of 10^-6; the figure describes higher ϵ as producing smoother softsign-like behavior that allows gradients to flow.Source: §3.3 · p. 4, Figure 3 · p. 4

3.4. Mean Centering

• Per-channel normalization avoids cross-channel interactions in filter updates, while TLU addresses the arbitrary activation bias introduced by omitting mean centering.The paper reports that uncentered normalization combined with TLU outperforms BN and other alternatives.Source: §3.4 · p. 5

3.5. Implementation and Practical Considerations

• FRN’s peak accuracy can depend on architecture-specific training choices: continuous cosine decay outperformed step decay, and warm-up helped sensitive models.A high initial learning rate can produce large early activations; learnable ϵ is also crucial for models with 1×1 activation maps.Source: §3.5 · p. 5

4.1. ImageNet Classification

• At the regular ImageNet batch size of 32 images/GPU, FRN outperforms BN and other methods on ResnetV2-50 and InceptionV3.This is described as the strongest BN baseline configuration; FRN also has higher training and validation accuracy than BN on ResnetV2-50.Source: Table 1 · p. 6, §4.1 · p. 6
• Across six ImageNet batch sizes, FRN consistently outperforms all tested normalization methods, while BN degrades at smaller sizes.GN is more consistent than BN, whereas Batch Renormalization degrades at smaller batch sizes.Source: Table 2 · p. 6, §4.1 · p. 6
• Ablations on ResnetV2-50 and InceptionV3 show complementary gains from FRN normalization and TLU activation.For ResnetV2-50, replacing IN with FRN gains 3.61 P@1 points, and adding TLU gains another 1.97 points.Source: §4.1 · p. 6, Table 3 · p. 7
• FRN outperforms other normalization methods even when applied after fully connected layers, a challenging case for per-sample normalization.The experiments include InceptionV3 and VGG-A; VGG-A has fully connected layers in its main network.Source: §4.1 · p. 7, Table 4 · p. 7
• For InceptionV3 and VGG-A, FRN required learning-rate warm-up and learned epsilon to achieve peak performance.Without warm-up, FRN underperformed other methods on InceptionV3 and failed to learn on VGG-A; the authors attribute this to max-pooling outputs saturating normalized activations and disrupting gradient flow.Source: §4.1 · p. 7
• Among tested TLU variants, a learnable threshold is needed for high performance with FRN, but it need not be input-dependent.Affine-TLU outperforms ReLU and PReLU in the reported comparison.Source: §4.1 · p. 7, Table 5 · p. 8

4.2. Object Detection on COCO

• FRN outperforms competing normalization methods at every tested batch size on ImageNet and COCO.On COCO, FRN beats BN by 0.9 AP points at the largest batch size and 8.9 AP points at the smallest, while exceeding GN throughout.Source: §4 · p. 5, §4.1 · p. 6, Table 6 · p. 8
• As COCO batch size falls to two images per GPU, BN drops 8.5 AP points, while FRN and GN each degrade by less than 0.6 AP points.The authors also report that FRN maintains a lead across metrics and batch sizes.Source: §4.2 · p. 8, Table 6 · p. 8

• FRN combines Filter Response Normalization with Thresholded activation (TLU), eliminating the need for batch-dependent training.Source: §5 · p. 8
• Across explored settings, FRN consistently performs strongly at large and small batch sizes, outperforming BN and Group Normalization.Source: §5 · p. 8
• FRN also succeeds in the pathological case of fully connected layers, which are typically not normalized.Source: §5 · p. 8
• Exploring FRN in other domains remains future work, including NLP, where Layer Normalization has been successful.Source: §5 · p. 8