Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures

• Deep CNNs have grown deeper and more complex, with the 2015 ImageNet winner using 152 layers and the runner-up using 22.The paper notes that this trend has also appeared in natural-language-processing benchmarks.Source: §1 · p. 2
• The computation and memory overhead of complex networks can limit the expansion of deep-learning applications.Some methods intended to improve learning may impose additional computation and memory overhead.Source: §1 · p. 2
• SimpleNet outperforms almost all deeper architectures while using 2 to 25 times fewer parameters.The authors designed it with minimal reliance on newer features and present it as a candidate for embedded devices.Source: §1 · p. 2

2.1 COMPLEX NETWORKS

• Complex-network work progressed from AlexNet to deeper VGGNet, GoogleNet, and ResNet designs, with methods including residual connections and stochastic depth supporting deeper models.The review also describes highway networks and wide residual networks as alternatives or extensions to deep architectures.Source: §2.1 · pp. 3–4

2.2 MODEL COMPRESSION

• Model-compression methods transfer or reproduce the behavior of larger models in less complex networks, including teacher–student distillation and FitNets’ intermediate hints.The review distinguishes compression approaches that create shallower, wider networks from FitNets’ thinner, deeper students.Source: §2.2 · p. 4

2.3 NETWORK PRUNING

• Deep Compression combines pruning, trained quantization, and Huffman coding, reducing AlexNet’s ImageNet storage from 240MB to 6.9MB without accuracy loss.The passage reports that the pipeline reduces storage by 35 to 49 times across networks.Source: §2.3 · p. 4

2.4 LIGHT WEIGHT ARCHITECTURES

• The paper proposes a 13-layer convolutional network that achieves state-of-the-art results on CIFAR10, with 2 to 25 times fewer parameters than previous deep architectures and superior or on-par performance.Source: §2.4 · p. 5

3 PROPOSED ARCHITECTURE AND DESIGN INTUITION

• SimpleNet uses 3 × 3 convolution kernels and 2 × 2 pooling operations in a homogeneous 13-layer design.The 11th and 12th layers use 1 × 1 kernels; the paper also describes batch normalization before ReLU and a dropout variant.Source: §3 · pp. 5–6
• Wide Residual Net later beat the authors’ record, while they report retaining the state-of-the-art result without zero-padding and normalization.The architecture is described as having the lowest number of parameters and operations among the Table 1 architectures, but the supplied text gives no values.Source: §3 · pp. 5–6, Table 1 · p. 6

3.0.1 GRADUAL EXPANSION AND MINIMUM ALLOCATION

• The authors recommend starting with a small, thin network and expanding it gradually and symmetrically toward a cone-shaped form.They describe progressively reducing spatial resolution while increasing feature-map richness, and caution against architectures that are extremely deep and thin or shallow and wide.Source: §3.0.1 · p. 6

3.0.2 HOMOGENEOUS GROUPS OF LAYERS

• The authors recommend organizing layers into homogeneous groups of gradually increasing width to manage parameter counts and provide information pools at semantic levels.Source: §3.0.2 · p. 6

3.1 LOCAL CORRELATION PRESERVATION

• Avoid 1 × 1 kernels in early layers because they ignore neighborhood information; the authors suggest using them near the network’s end or substituting 2 × 2 kernels.They note that 2 × 2 kernels can reduce parameters while retaining neighborhood information.Source: §3.1 · p. 6

3.2 MAXIMUM INFORMATION UTILIZATION

• Larger early feature maps and slower initial downsampling can make more information available without increasing network depth or parameter count.The authors state that, at the same depth and parameter count, networks using larger feature maps achieve higher accuracy.Source: §3.2 · p. 7

3.3 MAXIMUM PERFORMANCE UTILIZATION

• 2.7× faster training with 3 × 3 kernels using cuDNNv5 than cuDNNv4; smaller kernels also improve efficiency per parameter and can be cascaded for the same receptive field with more nonlinearity.The reported speedup is attributed to cuDNNv5; larger kernels impose overhead and can lose information.Source: Figure 2 · p. 7, §3.3 · p. 7

3.4 RAPID PROTOTYPING

• Test optimization policies before changing the architecture, then carefully tune the policy after the architecture is finalized.Learning rates and regularization need tuning; an automated policy can support quick initial tests.Source: §3.4 · pp. 7–8

3.5 EXPERIMENT ISOLATION

• Evaluate a feature under equal conditions, tracking prior changes and controlling discrepancies such as network entropy when comparing kernels.Otherwise, experimental changes may lead to incorrect deductions and inefficient designs.Source: §3.5 · p. 8

3.6 MINIMUM ENTROPY

• A model with fewer parameters that matches or exceeds heavier models is presented as evidence of better features and greater generalization, while greater parameter capacity can increase overfitting.The paper argues that constraining network entropy encourages reliance on more important, discriminative features.Source: §3.6 · p. 8

3.7 FINAL REGULATION STAGE

• Treat the guidelines as a starting point, then adjust one aspect at a time to find a performance–overhead compromise suited to the specific need.A systematic procedure helps identify which changes lead to particular outcomes.Source: §3.7 · p. 8
• For applications with medium or small datasets, heavy architectures bring computation, memory, and training-time overhead; lower overhead reduces time and power consumption, important for mobile applications.The passage frames this as a motivation for less expensive architectures that provide higher or comparable accuracy.Source: §3.7 · pp. 8–9

4 EXPERIMENTS

• Experiments cover five datasets and use one configuration across them, with fine-tuning only on CIFAR10.CIFAR10/100 used zero padding and mirroring; MNIST and SVHN used no data augmentation.Source: §4 · p. 9

4.1 CIFAR10/100

• 95.51% on CIFAR10 and 78.37% on CIFAR100 were achieved by Arch2, outperforming all major architectures except the 36m variant of WRN.Arch1 achieved a new state of the art on CIFAR10 without data augmentation.Source: §4.1 · p. 9, Table 2 · p. 9

4.2 MNIST

• 99.72% accuracy was achieved on MNIST with a 300K-parameter model, without data augmentation.The authors report second to state of the art without data augmentation and fine-tuning.Source: §4.2 · p. 10, Table 3 · p. 10

4.3 SVHN

• 2.37% test error was obtained on SVHN using the slimmed 300K-parameter version without data augmentation.Only the training and testing sets were used, not the extra training set.Source: §4.3 · p. 10, Table 4 · p. 10

4.4 EXTENDED TEST

• The test evaluates a slimmed version of the authors’ architecture with only 300K parameters and no dropout, comparing it with other deeper and heavier architectures with 2 to 20 times more parameters.Table 5 presents slimmed-version results on CIFAR10/100 datasets.Source: §4.4 · p. 10, Table 5 · p. 11

4.5 IMAGENET RESULTS

• On ImageNet, SimpleNet outperforms much deeper and larger architectures in the reported comparison.The experiment modifies downsampling for 224x224 inputs to balance information use against computation overhead.Source: Table 6 · p. 11, §4.5 · p. 11

5 CONCLUSION

• SimpleNet outperforms deeper and more complex architectures despite using considerably fewer parameters and operations.The authors present simplicity and design as a route to keeping computation in check while achieving high efficiency.Source: §5 · p. 12
• A 300K-parameter slimmed version also outperforms deeper or heavier architectures, according to the conclusion.The authors say it uses processing capacity efficiently.Source: §5 · p. 12
• Limited hardware constrained the authors to a few configurations, and they identify new applications and design choices as future work.They particularly note future experiments using recent advances in deep architectures.Source: §5 · p. 12

A.1 IMAGENET EXPERIMENT

• Delaying downsampling by one layer in m2 allows better use of larger feature maps and achieves higher accuracy than m1.m1 uses stride 2 in the first three layers; m2 uses stride 2 in layers 1, 2, and 4.Source: §A.1 · p. 16, Table 7 · p. 16
• The m1 variant is faster, while m2 achieves higher accuracy in the reported runtime-performance comparison.The appendix also presents an extended comparison of SimpleNet variants with larger and more complex architectures.Source: Table 7 · p. 16, Table 8 · p. 16, §A.1 · p. 16

A.2 EXTENDED RESULTS

• The appendix provides extended CIFAR10 and CIFAR100 results in Tables 9 and 10.It also flags that some results were achieved using several data-augmentation tricks.Source: §A.2 · pp. 16–17, Table 9 · p. 17, Table 10 · p. 17
• Table 11 compares ImageNet models by FLOPs and parameter count, while Inception v3/v4 size information comes from MXNet and TensorFlow, respectively.The appendix notes that Inception-ResNet-V2 would take 60 days of training with two Titan X GPUs to achieve the reported accuracy.Source: §A.2 · p. 17, Table 11 · p. 18

A.3 GENERALIZATION SAMPLES

• On unseen images, the early un-augmented SimpleNet correctly classifies examples despite their appearance differing from training images.The examples include airplanes with completely different appearances.Source: §A.3 · p. 18, Figure 3 · p. 18
• Figure 4 pairs deformed cat images with an animal drawing; its unusual predictions track prominent features, including a truck label linked to circular legs.The authors suggest these predictions reflect developed feature combinations.Source: §A.3 · p. 18, Figure 4 · p. 19
• The authors suggest limited model capability, CIFAR10 data without augmentation, or the optimization process could explain the remaining imperfections.They present these as possible explanations, not established causes.Source: §A.3 · p. 19
• A head/shoulder detector responds to unseen animal images but not a car, suggesting a possible cue for distinguishing animals from non-animals.The authors describe these results as encouraging but imperfect.Source: §A.3 · p. 19, Figure 5 · p. 20