Source-linked AI summary
FractalNet: Ultra-Deep Neural Networks without Residuals
Gustav Larsson, Michael Maire, Gregory Shakhnarovich
TL;DR
FractalNet asks whether residual learning is necessary for extremely deep neural networks. It uses self-similar architectures with interacting paths and drop-path regularization, matching residual networks on CIFAR and ImageNet while supporting anytime inference.
Problem
Residual learning had become central to extremely deep convolutional networks, motivating whether explicit residuals are necessary for competitive ultra-deep architectures.
Method
FractalNet repeatedly expands a simple self-similar macro-architecture into nested subpaths of different lengths, and uses drop-path to regularize their co-adaptation.
Results
FractalNet matches residual networks on CIFAR and ImageNet classification tasks, while drop-path reduces CIFAR-100 error from 35.34% to 28.20% without data augmentation.
Takeaways & Limitations
The experiments indicate that effectively shorter gradient-propagation paths during training may matter more than residual representations, while drop-path enables speed–accuracy tradeoffs through partial evaluation.
Abstract
from arXiv · showhide
We introduce a design strategy for neural network macro-architecture based on self-similarity. Repeated application of a simple expansion rule generates deep networks whose structural layouts are precisely truncated fractals. These networks contain interacting subpaths of different lengths, but do not include any pass-through or residual connections; every internal signal is transformed by a filter and nonlinearity before being seen by subsequent layers. In experiments, fractal networks match the excellent performance of standard residual networks on both CIFAR and ImageNet classification tasks, thereby demonstrating that residual representations may not be fundamental to the success of extremely deep convolutional neural networks. Rather, the key may be the ability to transition, during training, from effectively shallow to deep. We note similarities with student-teacher behavior and develop drop-path, a natural extension of dropout, to regularize co-adaptation of subpaths in fractal architectures. Such regularization allows extraction of high-performance fixed-depth subnetworks. Additionally, fractal networks exhibit an anytime property: shallow subnetworks provide a quick answer, while deeper subnetworks, with higher latency, provide a more accurate answer.
1 INTRODUCTION
FractalNet uses self-similar architecture to build competitive ultra-deep networks without residual connections. Its nested subpaths support implicit deep supervision, drop-path regularization, and anytime prediction.
- FractalNet constructs competitive extremely deep networks without relying on residual learning.
- A single final-layer loss and random initialization suffice for internal behavior that mimics deep supervision across subnetworks of many depths.The network can carve out useful assemblies of subnetworks when made sufficiently deep.
- A simple expansion rule creates nested subpaths of different depths, with join layers merging their feature blobs.The architecture uses a truncated fractal structure with intertwined columns and element-wise mean joins.
- Drop-path regularizes ultra-deep fractal networks by disabling macro-scale components while exploiting the architecture’s path structure.Without data augmentation, drop-path and dropout outperform residual networks regularized with stochastic depth.
- Drop-path can produce anytime behavior: shallow subnetworks return quick, moderately accurate predictions before the full network finishes.Its training schedule prevents subnetworks of different depths from co-adapting, requiring both shallow and deep subnetworks to produce correct outputs.
2 RELATED WORK
The related work frames extreme-depth training as an optimization and architecture problem. Prior approaches use nonlinearities, normalization, skip connections, stochastic depth, auxiliary losses, teacher-student training, and widening; FractalNet connects these ideas through effective depth and self-similarity.
- Increasing depth has improved convolutional-network performance but makes optimization harder for randomly initialized, highly nonlinear networks.The linear approximation underlying first-order gradient steps becomes increasingly inappropriate as depth and nonlinearity grow.
- ReLU activations, batch normalization, and improved initialization are prior optimization strategies intended to improve gradient propagation or training speed.
- Skip connections, highway networks, and ResNets use nontrivial routing, pass-through, or gating and are among the designs shown to scale to hundreds of layers.
- Stochastic depth and deep supervision suggest that reducing effective depth can help train extremely deep networks by providing shorter gradient paths.Stochastic depth randomly drops layers, while deep supervision adds an auxiliary loss at intermediate layers.
- Student-teacher methods couple networks of different depths by having a shallower student mimic a deeper teacher’s soft predictions.
- Widening architectures such as Inception combine outputs from layers with different receptive-field sizes, and fractal networks connect widening with increasing depth.
3 FRACTAL NETWORKS
FractalNet builds ultra-deep networks by recursively expanding self-similar blocks into multiple paths and joining transformed feature blobs without privileged residual or pass-through signals. Drop-path regularizes these paths, supports strong single-column subnetworks, and enables speed–accuracy choices across depths.
- Architecture: FractalNet recursively expands a base convolutional layer into self-similar blocks whose longest-path depth scales as 2^(C-1).Stacking the blocks with pooling layers B times yields total depth B · 2^(C-1).
- Architecture: Join layers merge feature blobs by element-wise mean, with every input produced by a preceding convolutional layer rather than a privileged pass-through channel.This distinguishes FractalNet’s joins from ResNet’s separation of pass-through and residual signals.
- Regularization via drop-path: Drop-path randomly disables parallel-path inputs while preserving at least one input-to-output path, preventing co-adaptation and discouraging anchor–corrective residual-like configurations.Local sampling drops join inputs independently; global sampling selects one path for the entire network, restricted to a single column.
- Regularization via drop-path: Alternating local and global drop-path sampling trains subnetworks of diverse path lengths and promotes individual columns as independently strong predictors.A new subnetwork is sampled each mini-batch, with experiments using a 50% local and 50% global mixture.
- Anytime behavior: Global drop-path also provides an anytime trade-off: shallow columns offer faster, moderately accurate predictions, while deeper columns offer greater accuracy at higher latency.Experiments show that high-performance single-column subnetworks can be extracted from the full fractal network.
- Experimental findings: FractalNet achieves competitive results without data augmentation, while drop-path and dropout outperform residual networks regularized with stochastic depth in the reported comparison.The paper also reports that increasing depth to 160 layers does not degrade FractalNet’s performance in the supporting experiments.
4 EXPERIMENTS
Experiments show that FractalNet scales across CIFAR, SVHN, and ImageNet, while drop-path improves regularization and supports high-performing subnetworks. Training within the fractal structure also mitigates depth-related convergence problems and produces student-teacher-like behavior.
- Ultra-deep scaling: Increasing fractal depth improves CIFAR-100 accuracy until diminishing returns, while FractalNet resists performance degradation at 160 layers.The supporting ultra-deep experiments use reduced feature channels and are not directly comparable to Table 1.
- CIFAR and SVHN: FractalNet depth-20 outperforms the original ResNet across CIFAR and achieves competitive SVHN performance, with results remaining consistent at depth 40.Without augmentation or regularization, FractalNet is superior to ResNet and ResNet with stochastic depth on CIFAR.
- Regularization: 28.20% CIFAR-100 error results from drop-path, down from 35.34% without regularization and below 44.76% for unregularized ResNet.ResNet with stochastic depth reaches 37.80% error in the same comparison.
- ImageNet: FractalNet matches ResNet at equal depth on ImageNet validation.The comparison uses the ImageNet validation set with 10-crop evaluation.
- Subnetworks: Drop-path trains extracted fixed-depth columns that outperform plain networks with the same topology, whose convergence degrades around 40 layers and fails at 160 layers.The deepest column performs close to the full network, suggesting fractal structure may function more as a training framework than a final architecture.
- Training dynamics: A 40-layer fractal column improves after about 25 epochs as shorter subnetworks stabilize, unlike a plain 40-layer network that never makes fast progress.Drop-path pressures the deep column to adapt using the rest of the network as a teacher.
- Interpretation: A single extracted FractalNet column performs nearly as well as the full network without ensembling, providing a counterexample to ensemble-like explanations of residual-network success.On CIFAR-10, the deepest column is statistically equivalent to the full network.
5 CONCLUSION
The conclusion argues that effective path length, rather than residual representations, is central to training ultra-deep networks. FractalNet and drop-path provide a simple architecture and regularization scheme that also support latency–accuracy tradeoffs and reveal internal deep-supervision-like behavior.
- 5 CONCLUSION: Experiments suggest that path length is fundamental for training ultra-deep networks, while residuals are incidental.FractalNet and ResNet share large nominal depth with effectively shorter gradient-propagation paths.
- 5 CONCLUSION: Drop-path regularizes deep fractal networks and enables speed–accuracy tradeoffs through partial evaluation.Shallow subnetworks can provide useful answers before the full network finishes.
- 5 CONCLUSION: FractalNet’s internal behavior resembles engineered modules, deep supervision, and student-teacher learning found in other network designs.These connections emerge from the architecture’s substructure and training evolution.