Source-linked AI summary
Exploring Randomly Wired Neural Networks for Image Recognition
Saining Xie, Alexander Kirillov, Ross Girshick, Kaiming He
TL;DR
Existing neural architecture search methods optimize within wiring spaces that remain constrained by hand-designed network generators. The paper defines stochastic network generators, uses three classical random graph families to create randomly wired networks, and finds several variants with competitive ImageNet accuracy, motivating further generator design.
Problem
NAS searches wiring and operations within a network-generator-defined space whose allowed wiring patterns remain hand designed and constrained.
Method
The paper defines network generators and stochastic generators, then uses classical random graph models to produce randomly wired neural networks.
Results
Several random-generator variants achieve competitive ImageNet accuracy, with the best Watts-Strogatz generators outperforming or matching manually designed and NAS-found networks.
Takeaways & Limitations
The findings suggest that designing network generators can explore less constrained families of models beyond fixed NAS generators.
Takeaways & Limitations
Randomly wired networks still encode human-designed priors through generator rules and use an arbitrary, human-designed mapping from graphs to neural operations.
Abstract
from arXiv · showhide
Neural networks for image recognition have evolved through extensive manual design from simple chain-like models to structures with multiple wiring paths. The success of ResNets and DenseNets is due in large part to their innovative wiring plans. Now, neural architecture search (NAS) studies are exploring the joint optimization of wiring and operation types, however, the space of possible wirings is constrained and still driven by manual design despite being searched. In this paper, we explore a more diverse set of connectivity patterns through the lens of randomly wired neural networks. To do this, we first define the concept of a stochastic network generator that encapsulates the entire network generation process. Encapsulation provides a unified view of NAS and randomly wired networks. Then, we use three classical random graph models to generate randomly wired graphs for networks. The results are surprising: several variants of these random generators yield network instances that have competitive accuracy on the ImageNet benchmark. These results suggest that new efforts focusing on designing better network generators may lead to new breakthroughs by exploring less constrained search spaces with more room for novel design.
1. Introduction
Network wiring is a central design factor in image recognition, but NAS still searches within manually constrained wiring spaces. This paper studies stochastic network generators based on classical random graphs and finds several competitive ImageNet networks.
- ResNet and DenseNet demonstrate that elaborate connectivity patterns contribute substantially to their effectiveness.
- NAS jointly searches wiring patterns and operations, but its network generator remains hand designed and restricts the allowed wiring patterns.
- The paper uses stochastic network generators based on Erdős-Rényi, Barabási-Albert, and Watts-Strogatz random graph models.
- Several random-generator variants achieve competitive ImageNet accuracy, with Watts-Strogatz generators producing networks that outperform or match manual and NAS-designed counterparts.
- Randomly wired networks are not prior-free because generator rules and graph-model distributions encode structural preferences such as clustering.
- The paper proposes shifting attention from designing individual networks to designing generators that produce families of models for search.
2. Related Work
Related work traces neural-network wiring from early chain-like and randomly connected machines to structured modern architectures, NAS, and graph-theoretic random models. These traditions motivate studying connectivity as both a design choice and a modeled graph property.
- Network wiring: Early RNNs and CNNs used chain-like wiring, while LSTMs, Inception, ResNets, and DenseNets introduced increasingly structured or branching connectivity.
- Neural architecture search (NAS): NAS research has expanded its optimization methods, but the underlying search space defined by the NAS network generator has remained largely unchanged.
- Randomly wired machines: Early randomly wired machines included Turing’s unorganized machines, Minsky’s randomly wired learning machine, and Rosenblatt’s randomly connected perceptron.
- Relation to neuroscience: Neuroscience observations connect randomly organized neural structures with naturally varying physical connections and small-world properties in nematode neural networks.
- Random graphs in graph theory: Random graph models encode different probabilistic behaviors and prior knowledge, making them useful for modeling and analyzing real-world graphs.
3. Methodology
The methodology reframes architecture design around network generators, including stochastic generators that sample architectures from a parameterized distribution. It then constructs randomly wired networks by mapping classical random graphs into neural-network computation graphs while minimizing additional optimization.
- Network Generators: A network generator maps parameters θ to a family of neural network architectures, while stochastic generators additionally use a seed s to sample varied instances.The generated representation specifies operations and data flow; learned network weights are acquired after generation.
- NAS Perspective: The paper interprets NAS as stochastic generation in which an LSTM produces probability distributions, seeds sample construction actions, and reinforcement learning optimizes controller parameters θ.Hand-designed rules also map sampled actions into a computational DAG and therefore remain part of the NAS generator.
- Network Generators: Encapsulating the complete generation process exposes which design choices are optimized versus hard-coded, revealing that NAS searches within a manually restricted network space.The paper notes that standard NAS cell rules impose fixed input and output degrees on nodes, excluding many possible graphs.
- Randomly Wired Neural Networks: Nodes aggregate incoming data, apply a ReLU-convolution-BN transformation, and distribute the result along outgoing edges, while directed edges represent tensor data flow.The illustrated node uses a weighted sum with learnable positive weights and sends transformed data to multiple outputs.
- Randomly Wired Neural Networks: Randomly wired networks are generated by sampling classical ER, BA, or WS graph models, converting each graph into a DAG, and applying a simple mapping to neural-network roles.The mapping is intentionally simple so the study can focus on graph wiring patterns, although it remains human-designed.
- Design and Optimization: The study performs no random seed search because accuracy varies little across seeds, using only minimal one- or two-parameter grid search beyond hand-coded generator design.Random search remains possible for any stochastic generator, but the paper suggests its benefit may be small for these networks.
4. Experiments
Experiments evaluate randomly wired networks across ImageNet regimes and analyze generator, graph-damage, and node-operation effects. RandWire achieves competitive accuracy, while generator choice and wiring influence performance and robustness.
- Experimental setup: ImageNet experiments span small and regular computation regimes, with RandWire complexity controlled through node count N and channel count C.Networks use N=32; C is adjusted to meet target complexity.
- Random generators: 73.8% mean accuracy is achieved by WS(4, 0.75), while ER, BA, and WS settings each exceed 73% within a <1% gap.All random generators converge across five sampled network instances.
- Random generators: 0.2–0.4% standard deviation across most generators indicates low variation among random network instances, despite accuracy gaps between generators.A BA(1) generator reaches 70.7%, approximately 3% below WS(4, 0.75).
- Analysis experiments: WS networks lose more accuracy when higher-output-degree nodes are removed, whereas edge-removal loss generally decreases with the target node’s input degree.ER networks are less sensitive to edge removal, possibly because each edge is wired independently.
- Analysis experiments: Replacing node operations preserves generator accuracy rankings, with Pearson correlations of 0.91–0.98 between operation-specific accuracy series.Even 3×3 pooling followed by 1×1 convolution performs similarly to 3×3 convolution.
- Comparisons: 1.9% and 1.3% higher mean accuracies than ResNet-50 and ResNet-101, respectively, are reported under comparable FLOPs; gains over ResNeXt are 0.6%.In the larger-computation comparison, RandWire is 0.7%–1.3% below the most accurate NAS models while using approximately 2/3 the FLOPs and 3/4 the parameters.
- Comparisons: RandWire features also transfer to object detection, following a trend similar to the ImageNet comparisons.The passage attributes this result to features learned by randomly wired networks.
5. Conclusion
The paper explores randomly wired neural networks generated from classical random graph models. Their mean ImageNet accuracy is competitive with hand-designed and NAS-optimized models, motivating further network-generator design.
- Conclusion: Three classical random graph models drive randomly wired neural networks through the paper’s network-generator framework.The conclusion identifies the network generator as enabling this exploration.
- Conclusion: Competitive mean ImageNet accuracy with hand-designed and NAS-optimized models motivates exploring new generator designs.The stated future direction is to seek new, powerful network designs through generator exploration.
A. Appendix
The appendix describes how NAS cells can be represented as graphs and how undirected random graphs are converted into directed acyclic networks. These representations expose structural priors while ensuring acyclic data flow.
- Mapping a NAS cell to a graph: Mapping NAS combining operations to nodes and unary transformations to edges reveals that internal nodes have input degree 2 and output degree 1.This property is an implicit prior induced by the NAS search-space design.
- Mapping a NAS cell to a graph: An alternative mapping places both combining and unary transformations on nodes, with data flow represented by edges.Under this mapping, internal merging nodes retain input degree 2 and output degree 1.
- Converting undirected graphs into DAGs: ER, BA, and WS undirected graphs become DAGs by directing every edge from the lower-indexed node to the higher-indexed node.The indexing heuristic guarantees that the resulting directed graph contains no cycle.