Source-linked AI summary
Growing a Neural Network in Breadth, Depth, and Time
Eivinas Butkus, Kedar Garzón Gupta, Nikolaus Kriegeskorte
TL;DR
Biological and artificial intelligence operate under spatial and temporal resource constraints, but how these resources jointly shape computational architectures remains a central design question. This paper introduces differentiable costs for breadth, depth, and time, optimizes them with task error, and finds that resources trade off, grow with task complexity, and produce adaptive processing dynamics correlated with human reaction times.
Problem
Understanding intelligent systems requires accounting for how spatial and temporal resource constraints shape the architectures that solve tasks.
Method
The paper jointly optimizes differentiable breadth, depth, and time costs with task errors via backpropagation in a recurrent convolutional network.
Results
Breadth, depth, and time are fungible, networks grow across all three dimensions with task complexity, and adaptive recurrent steps emerge for occluded inputs while correlating with human reaction times.
Takeaways & Limitations
Resource optimization can generate diverse computational graphs and human-like processing dynamics without training on human data.
Takeaways & Limitations
The networks are small-to-intermediate, resource interactions may not be captured by a linear loss, and individual adaptive-resource methods are not benchmarked.
Abstract
from arXiv · showhide
Spatial and temporal resource constraints are critical for both biological and artificial intelligent systems. Here we define differentiable cost terms for breadth, depth, and time within a recurrent convolutional neural network conceived as a finite subset of an infinite lattice. We optimize these costs jointly with task errors via backpropagation. We set different pressures on breadth, depth, and time, which leads to diverse computational graphs emerging organically through training. We find that all three resources can be traded off against each other to achieve a given level of accuracy. Networks grow in all three dimensions with task complexity and spontaneously take more recurrent steps when inputs are occluded. Surprisingly, time used by the model correlates with human reaction times in an object recognition task. Our framework provides a normative account of how resource constraints shape neural architectures, connecting to questions about brain design in neuroscience, and may help illuminate the diversity of neural solutions found in nature.
1 Introduction
The paper introduces a framework that jointly prices breadth, depth, and time while training recurrent convolutional networks, allowing task demands and resource pressures to shape computational graphs. It finds that these resources can substitute for one another and that adaptive processing time tracks input difficulty and human reaction times.
- Motivation: Intelligence is framed as achieving goals under spatial and temporal resource constraints, motivating explicit costs for neural architectures.The introduction connects these pressures to metabolic, wiring, speed, compression, distillation, and architecture-design concerns.
- Gap and approach: Prior work treats breadth, depth, and time separately, whereas this framework optimizes all three jointly within one model.Earlier methods include post-hoc pruning, differentiable channel sparsity, and adaptive computation time.
- Framework: Differentiable resource costs are optimized alongside task errors, enabling each network to select a resource profile and grow organically across breadth, depth, and time.The architecture is represented as a finite computational graph positioned within an infinite lattice of possible resource use.
- Framework: The model uses a recurrent convolutional architecture in which breadth corresponds to channels, depth to layers, and time to recurrent processing steps.Resource pressure prunes channels, layers, and time steps, producing a compact subgraph from the full architecture.
- Findings: Breadth, depth, and time are fungible: networks with different resource profiles can achieve comparable accuracy, and all three dimensions expand as task complexity increases.The paper reports shallow-wide and deep-narrow networks matching accuracy, with more resources used on more complex datasets.
- Findings: Adaptive time allocation emerges organically: models take more recurrent steps for occluded inputs, and image-wise processing time correlates with human reaction times.This association arises despite the models never being trained on human data.
2 Related work
The paper situates multi-resource optimization within cognitive resource-rationality research, neural wiring and recurrent-processing studies, and deep-learning methods for adaptive architectures. It distinguishes its approach by jointly applying differentiable costs for breadth, depth, and time within one computational framework.
- Cognitive resource constraints: Resource-rationality research studies how cognitive limits constrain symbolic, probabilistic, and computational agents.The paper places its neural resource constraints alongside bounded rationality, computational rationality, and resource rationality.
- Computational neuroscience: Neuroscience work examines spatial wiring economy and temporal recurrence, including links between recurrent processing and human object-recognition dynamics.The paper extends these lines by jointly considering spatial and temporal resource dimensions.
- Deep learning: Deep-learning methods have separately trained models with adaptive depth, breadth, or time, while pruning often removes units after training through iterative removal and fine-tuning.Neural architecture search instead explores discrete architectural choices, including differentiable operation-selection methods.
- Positioning: The paper claims the first joint optimization of differentiable breadth, depth, and time costs within a single framework.This contrasts with prior methods that address individual resources or search over discrete architectural choices.
3 Methods
The method jointly optimizes task error with differentiable costs for breadth, depth, and time, allowing recurrent convolutional networks to form resource-constrained computational graphs. Experiments vary resource pressures and task difficulty while measuring effective network usage through post-hoc pruning.
- Multi-resource optimization: The loss combines task error with differentiable breadth, depth, and time costs, whose relative prices are controlled by λ coefficients.λ_errors is fixed at 1 while breadth, depth, and time pressures are varied.
- Model: Breadth corresponds to channels, depth to layers, and time to recurrent processing steps in a finite computational graph representing an infinite resource lattice.The recurrent convolutional network uses bottom-up, lateral, and top-down connections across layers and time.
- Time selection: Adaptive time selection computes input-dependent weights over per-timestep logits, whereas fixed selection applies one learned temporal allocation uniformly across inputs.The final prediction is a weighted mixture of the per-timestep probability distributions.
- Resource costs: Breadth is penalized by channel rank, depth by layer index, and time by the expected normalized timestep under the time-selection weights.For adaptive selection, the time cost varies per input and can favor additional processing when its error reduction outweighs its cost.
- Resource measurement: Effective resource usage is measured post hoc by finding the smallest sub-network that preserves 98% of above-chance accuracy, then counting surviving layers, channels, and expected time.Depth counts active layers, breadth averages surviving channels per active layer, and time uses the expected timestep index.
- Experimental design: Experiments use CIFAR-10 and vary breadth, depth, and time pressures jointly, with matched pressure settings across MNIST, CIFAR-10, and Tiny ImageNet to test task-complexity effects.Error normalization by log K makes comparisons across datasets with different class counts possible.
4 Results
Jointly varying pressures on breadth, depth, and time produces sparse, diverse networks that trade resources against accuracy. The models adapt computation to input difficulty, align some timing behavior with humans, and grow spatial resources as tasks become more complex.
- Breadth vs. depth: Increasing λbreadth and λdepth smoothly reduces raw and pruning-defined resource costs, yielding sparse subnetworks that preserve 98% of above-chance accuracy without fine-tuning.Weights concentrate into fewer channels and layers as resource pressure increases.
- Breadth vs. depth: Shallow-and-wide models can achieve comparable accuracy to narrow-and-deep models, showing that breadth and depth are partially fungible at a given performance level.Accuracy nevertheless decreases as overall breadth and depth pressure increase.
- Breadth vs. depth: More constrained models rely on diffuse, low-level image features, whereas less constrained models attend to localized high-level features such as faces or tires.Attribution-map entropy is higher in constrained models, but its strong correlation with accuracy makes depth effects difficult to disentangle.
- Time: Adaptive time selection achieves higher accuracy than fixed selection at every level of time used, while increasing λtime reduces both time cost and time used.Adaptive allocation therefore dominates assigning the same temporal budget to every input.
- Time: Occlusion increases adaptive computation time, and ambiguous or atypical images receive more processing than canonical, easy-to-classify examples.Model time used is significantly correlated with human reaction times (ρ = 0.299, p < 0.001) and human judgment uncertainty.
- Breadth vs. depth vs. time: Pareto-optimal models achieving at least 70% accuracy span breadth, depth, and time, indicating that all three resources can compensate for one another.Depth also changes solution strategy: shallow and deep models make qualitatively different errors, while task complexity drives denser networks and greater spatial resource use.
5 Discussion
The framework shows that breadth, depth, and time can be traded off and grow with task complexity, while adaptive computation produces human-like timing patterns. Its scope is limited by small model scale, a linear cost combination, and the absence of single-resource benchmarks.
- Breadth, depth, and time can be traded against one another and grow organically with task complexity.
- Adaptive time allocation emerges under time pressure and correlates with human reaction times, suggesting human-like processing dynamics from resource optimization alone.
- The networks are small-to-intermediate, so whether these trade-offs hold for larger models remains untested.
- The linear loss combination may miss resource interactions, and the study does not benchmark adaptive breadth, depth, or time methods individually.
- The framework is extensible to costs such as energy and data efficiency, and may help study neural architecture diversity across ecological niches.
A Compute
The study used substantial but distributed GPU compute to train its reported model set.
- 1,273 models required approximately 3,200 GPU-hours in total.Each model was trained on a single GPU for approximately 2.5 hours and used roughly 3.3 GB of GPU memory at batch size 128.
- Training used a university cluster containing NVIDIA GeForce RTX 2080 Ti, A40, and L40 GPUs.
- Additional compute was used during model development and preliminary experiments but was not formally tracked.
B Experiments
Table 1 summarizes the experimental configurations and identifies how the breadth-versus-depth analysis was selected from the broader three-way experiment.
- Total models equals the product of all variable levels and the number of instances.
- The breadth-versus-depth results use the λtime = 0 slice of the three-way experiment.
- Table 1 therefore distinguishes the full experimental configuration space from the subset used for breadth-versus-depth results.
C Noise
The model adds Gaussian hidden-state noise during recurrent processing to prevent unconstrained weight scaling from eliminating breadth and depth costs without harming performance.
- Gaussian noise with σ = 0.1 is added to hidden states at each time step.
- The noise establishes a floor below which weights cannot be reduced without degrading signal-to-noise ratio.
- Without internal noise, an artificial network could scale down all weights to minimize breadth and depth costs without affecting performance.
D Annealing Noise and Resource Costs
Training begins without resource costs so the network can learn representations from task error alone. Costs and noise are then introduced through linear schedules.
- Resource costs are set to zero during warmup, allowing representation learning under the error cost alone.Warmup lasts 15 epochs for space costs and 20 epochs for time.
- Space and time costs are linearly annealed to their full values over 10 epochs.
- Noise increases linearly from zero to σ = 0.1 over the first 15 epochs.
E Training Details
Models are trained for 150 epochs with AdamW and mixed precision. Optimization uses a warmup-hold-cosine learning-rate schedule with gradient clipping.
- Models are trained for 150 epochs using AdamW with lr = 10^-3, β = (0.9, 0.999), and weight decay = 0.1.Training uses mixed precision.
- The learning rate warms up for 5 epochs, stays constant for 25 epochs, then decays cosinely to 10^-6 over 120 epochs.Costs are fully active before decay begins.
- Gradients are clipped to a maximum norm of 1.0.
F Pruning Algorithm
The pruning algorithm identifies compact subnetworks that preserve 98% of above-chance accuracy, then converts their surviving channels into effective breadth and depth measures. Attribution maps are estimated separately by localized image perturbations.
- Pruning Algorithm: The procedure uses iterative binary search to find the smallest subnetwork preserving 98% of the model’s above-chance accuracy.
- Pruning Algorithm: Channels are ranked by average weight magnitude, then binary search zeroes channels below a norm cutoff while evaluating accuracy on 10,000 test images.Thirty log-space iterations adjust the cutoff to balance pruning against threshold preservation.
- Resource Definitions: The final binary mask defines depth as active layers and breadth as the average surviving channels per active layer.
- Attribution Analysis: Attribution maps are computed by adding Gaussian noise within sliding 5 × 5 pixel windows and measuring the correct-class probability drop.Each pixel receives the average score across patches containing it.