Source-linked AI summary
Convolutional Neural Fabrics
Shreyas Saxena, Jakob Verbeek
TL;DR
CNN architecture selection spans an exponentially large discrete space, making systematic exploration difficult. The paper introduces a locally connected three-dimensional fabric that embeds architectures and ensembles them through shared weights, achieving competitive results across image classification and semantic segmentation benchmarks.
Problem
CNN architecture design involves an exponentially large space of choices, but efficient systematic exploration of that space remains lacking.
Method
A fabric uses a three-dimensional trellis of response maps with local layer, scale, and channel connections, allowing paths or ensembles of architectures to share weights and be learned by back-propagation.
Results
Results are close to the best reported literature results on MNIST, CIFAR10, and Part Labels, including 95.6% super-pixel accuracy on Part Labels.
Takeaways & Limitations
Fabrics can sidestep explicit selection, training, and testing of individual architectures while remaining competitive with hand-crafted CNNs across classification and segmentation tasks.
Takeaways & Limitations
Channel-doubling fabrics are most useful with sparse connectivity because dense connectivity causes explosive parameter growth, and experiments were left for future work.
Abstract
from arXiv · showhide
Despite the success of CNNs, selecting the optimal architecture for a given task remains an open problem. Instead of aiming to select a single optimal architecture, we propose a "fabric" that embeds an exponentially large number of architectures. The fabric consists of a 3D trellis that connects response maps at different layers, scales, and channels with a sparse homogeneous local connectivity pattern. The only hyper-parameters of a fabric are the number of channels and layers. While individual architectures can be recovered as paths, the fabric can in addition ensemble all embedded architectures together, sharing their weights where their paths overlap. Parameters can be learned using standard methods based on back-propagation, at a cost that scales linearly in the fabric size. We present benchmark results competitive with the state of the art for image classification on MNIST and CIFAR10, and for semantic segmentation on the Part Labels dataset.
1 Introduction
Convolutional neural fabrics address CNN architecture selection by embedding many architectures in one trellis rather than choosing and training them individually. Their shared-weight paths can represent single architectures or ensembles while retaining efficient back-propagation-based learning.
- Motivation: CNN progress is hindered by the lack of efficient systematic methods for exploring an exponentially large architecture space.The space includes choices such as depth, channels, filters, pooling, connectivity, and activation types.
- Fabric: A fabric is a three-dimensional trellis of response maps at different resolutions, with local connections across layers, scales, and channels.Each activation combines a multidimensional neighborhood from the previous layer using a linear function followed by a non-linearity.
- Architecture space: The fabric embeds architectures arising from varied choices of the ten CNN hyper-parameters, while exposing only layer count and channel count as main hyper-parameters.These two settings are not critical when the fabric is sufficiently large.
- Learning: Single architectures are recovered by activating one path, whereas general weight settings ensemble many architectures that share parameters where their paths overlap.The acyclic trellis supports standard error back-propagation for configuring individual architectures or ensembles.
- Contributions: Fabrics scale linearly in computation and memory while leveraging exponentially many chain-structured architectures through extensive weight sharing.Their multi-scale structure also supports outputs at multiple resolutions within one non-branching network.
2 Related work
Prior CNN architectures are effective but are commonly selected by trying models individually, while related multi-resolution and stochastic approaches do not provide the same broad architecture-spanning role. The paper positions fabrics as a structure that spans large classes of classification and dense-prediction networks.
- CNN architecture selection: Widely used architectures such as AlexNet and VGG are effective, but their optimality under computational and memory constraints remains unclear.Their adoption is attributed in part to the lack of better methods than testing architectures one by one.
- Structured prediction: Semantic-segmentation architectures are often derived from classification networks and use up-sampling to recover resolution lost through pooling.Other designs add same-resolution links or process inputs across several resolutions in parallel.
- Positioning: Earlier multi-dimensional networks had not, to the authors’ knowledge, been explored as a basis for spanning large classes of convolutional neural networks.This frames fabrics as an architecture-space representation rather than a single fixed network.
- Related architectures: Related cross-stitch and interlinked CNNs exchange information across architectures or input resolutions but do not address architecture selection in the same way.The cited interlinked structure also lacks sparse channel connectivity and manually sets filter sizes per node.
- Paper contribution: Fabrics extend a similar interlinked structure to span a vast class of networks for both dense prediction and classification.The stated contribution covers both task families rather than only semantic segmentation.
- Stochastic methods: Unlike dropout and swapout, which approximately average variations of one architecture, fabrics leverage an exponentially large class through continuous optimization.The approaches are described as orthogonal and applicable together.
3 The fabric of convolutional neural networks
The fabric is an acyclic, three-axis trellis whose local operations can reproduce common CNN design choices and whose paths or activated subgraphs represent different networks. Its sparse construction keeps learning memory and parameter growth substantially lower than dense alternatives, while channel doubling trades modest activation cost for many more channels.
- Fabric structure: Each fabric node is a response map organized along layer, scale, and channel axes.Layers enforce acyclicity, scales represent resolutions separated by factors of two, and channels group maps at each scale and layer.
- Connectivity: Nodes use a sparse homogeneous 3×3 scale-channel neighborhood from the previous layer, with strided convolution for fine-to-coarse input and upsampling followed by convolution for coarse-to-fine input.For 2D images, activations depend on a four-dimensional 3×3×3×3 neighborhood, while convolution applies only across input dimensions.
- Re-sampling operators: Fabrics can implement re-sampling choices including stride-two convolution, average pooling, and coarse-to-fine up-sampling.Larger receptive fields or strides are obtained by repeating the corresponding operations.
- Re-sampling operators: ReLU units and strided convolutions can implement 2×2 max-pooling by computing signed combinations, applying ReLU, aggregating, and repeating across directions.The construction produces each pixel’s maximum with its vertical neighbor before horizontal repetition and subsampling.
- Embedded architectures: Chain-structured networks correspond to paths formed by zeroing off-path weights, while activating larger edge subsets can represent non-chain networks.The trellis embeds more than chain structures by distributing subsampling across layers and including additional up-sampling or branching patterns.
- Channel connectivity: Sparse channel connectivity can emulate dense convolutional layers by copying channels, convolving them, and locally aggregating the results.Copy and sum operations use local channel interactions and identity or zero filters.
- Complexity: Activation memory grows linearly in channels and layers despite exponentially many embedded architectures, while sparse parameters grow linearly in channels and dense parameters quadratically.Across scales, total input-resolution elements are bounded by 2^D/(2^D − 1) times the input-resolution count.
- Complexity: For 32×32 inputs, a 16-layer, 256-channel fabric has 2M sparse parameters and 6M activations, versus 170M parameters when densely connected.For 256×256 inputs, a 16-layer, 64-channel fabric has 0.7M sparse parameters, 16M dense parameters, and 89M activations.
4 Experimental evaluation results
Experiments evaluate sparse and dense fabrics across MNIST, CIFAR10, and Part Labels, finding competitive performance and generally noncritical layer and channel choices when trellises are sufficiently large.
- Datasets and training: Experiments cover image classification on MNIST and CIFAR10 and semantic segmentation on Part Labels.The study uses standard dataset splits and task-specific augmentation protocols.
- Model settings: Larger trellises perform comparably or better than smaller ones, making the number of layers and channels noncritical when capacity is sufficient.Sparse and dense fabrics with varied sizes were evaluated on all three datasets.
- Part Labels: 95.6% super-pixel accuracy is obtained on Part Labels with both sparse and dense trellises.The sparse trellis uses roughly 4,000× fewer parameters than VGG-16-based alternatives and trains from about 2,000 images without spatial-prior models.
- MNIST: 0.48% and 0.33% error rates are obtained on MNIST with sparse and dense fabrics, respectively.The sparse model uses 20× fewer parameters than the dense variant, while the dense result is comparable to work using similar augmentation.
- CIFAR10: 7.43% error is obtained with a dense fabric on CIFAR10, while the sparse fabric reaches 18.89%.The sparse model’s weaker performance may reflect insufficient capacity or optimization difficulty.
- Learned connectivity: Pruning 67% of CIFAR10 connections raises error from 7.4% to 8.1% after fine-tuning, and removes all up-sampling connections.The visualization shows different learned scale-processing patterns for segmentation and classification fabrics.
5 Conclusion
The conclusion presents convolutional neural fabrics as locally connected trellises that embed many convolutional networks while reducing the need to specify and test architectures individually. Experiments suggest competitive performance, and ongoing work extends the fabric design and tasks.
- Conclusion: Convolutional neural fabrics are homogeneous, locally connected trellises over response maps that subsume a large class of convolutional networks.They use layers and channels as their two main hyper-parameters.
- Conclusion: Fabrics avoid specifying, training, and testing individual architectures one by one to find a suitable network.The approach uses more resources than one embedded architecture but costs less than testing all embedded architectures separately.
- Conclusion: Experiments on MNIST, CIFAR10, and Part Labels obtain results close to the best reported literature results across classification and segmentation.The authors note that this sometimes uses more parameters, but uses far fewer on Part Labels.
- Future work: Ongoing work explores channel-doubling fabrics, joint classification, detection and segmentation, intermediate channel connectivity, and scale-axis convolution.These directions target broader task combinations and improved cross-scale processing.
A Supplementary Material
The supplementary material visualizes connection strengths in learned fabrics for Part Labels, MNIST, and CIFAR10, with layers arranged horizontally and scales vertically.
- Supplementary visualizations: Connection-strength visualizations compare learned fabrics for Part Labels, MNIST, and CIFAR10.Layers are laid out horizontally, while scales are laid out vertically.
A.1 Fabric visualizations
Learned fabrics configure distinct connection patterns for segmentation and classification, and even distinguish between MNIST and CIFAR10. Pruning can remove most connections while retaining relatively low error.
- Learned connection patterns differ between segmentation and classification because the tasks require different output resolutions.
- MNIST uses a band-diagonal propagation pattern across multiple scales, while CIFAR10 processes the finest scale first before propagating downward.
- After pruning 67% of fabric connections and fine-tuning, CIFAR10 error is 8.1% versus 7.4% for the full fabric.
A.2 Fabric structure
Fabric nodes combine signals across scales through resolution-adapting convolutions and ReLU activation, while sparse channel connectivity forms a 3D local trellis.
- Finer-scale inputs use strided convolution, same-scale inputs use normal convolution, and coarser-scale inputs are zero-padding upsampled before convolution.
- The three transformed inputs are added and passed through a ReLU activation.
- In the 3D fabric, each internal node is one response map at a given resolution and receives inputs from 3 × 3 scale-channel-plane nodes in the previous layer.
- Each edge represents a 3 × 3 spatial convolution, producing a 4D input tensor spanning two spatial dimensions, scale, and channel.
A.3 Additional experimental results
Additional evaluations cover segmentation and classification with sparse and dense fabrics, using test-set measurements and varying fabric sizes. Larger fabrics generally perform better or comparably under simple regularization.
- Additional Part Labels results include superpixel- and pixel-level segmentations, with tables reporting sparse and dense fabrics of different sizes and parameter counts.
- The reported results use test sets with models trained only on training data, whereas the main paper also trains on validation data.
- Larger fabrics generally achieve better or comparable results than smaller fabrics across all three datasets despite weight decay and early stopping.
- The supplementary tables report MNIST and CIFAR10 error rates plus Part Labels accuracies for sparse and dense CNF variants.