Source-linked AI summary
Adaptive recurrent vision performs zero-shot computation scaling to unseen difficulty levels
Vijay Veerabadran, Srinivas Ravishankar, Yuan Tang, Ritik Raina, Virginia R. de Sa
TL;DR
The paper asks whether adaptive computation lets recurrent vision models scale processing beyond training difficulty, a capability insufficiently tested on challenging visual tasks. It combines ConvRNNs with Graves’ (2016) learned halting mechanism and evaluates several recurrent architectures on PathFinder and Mazes. AdRNNs adjust iterations to image difficulty and generalize zero-shot to harder unseen instances by increasing recurrent computation.
Problem
Whether adaptive recurrent computation enables vision models to generalize zero-shot to harder difficulty levels remained insufficiently studied beyond simple tasks and language.
Method
The paper combines ConvRNNs with Graves’ (2016) learnable halting mechanism and evaluates recurrent architectures including LocRNN on PathFinder and Mazes.
Results
AdRNNs dynamically halt earlier or later according to problem difficulty and increase recurrent iterations to generalize zero-shot to harder unseen settings.
Takeaways & Limitations
Adaptive recurrent processing provides modeling evidence for allocating computation conditionally on visual-input requirements while extrapolating to harder reasoning problems.
Takeaways & Limitations
The study applies only to static images, leaving extension to time-varying video input for future work.
Abstract
from arXiv · showhide
Humans solving algorithmic (or) reasoning problems typically exhibit solution times that grow as a function of problem difficulty. Adaptive recurrent neural networks have been shown to exhibit this property for various language-processing tasks. However, little work has been performed to assess whether such adaptive computation can also enable vision models to extrapolate solutions beyond their training distribution's difficulty level, with prior work focusing on very simple tasks. In this study, we investigate a critical functional role of such adaptive processing using recurrent neural networks: to dynamically scale computational resources conditional on input requirements that allow for zero-shot generalization to novel difficulty levels not seen during training using two challenging visual reasoning tasks: PathFinder and Mazes. We combine convolutional recurrent neural networks (ConvRNNs) with a learnable halting mechanism based on Graves (2016). We explore various implementations of such adaptive ConvRNNs (AdRNNs) ranging from tying weights across layers to more sophisticated biologically inspired recurrent networks that possess lateral connections and gating. We show that 1) AdRNNs learn to dynamically halt processing early (or late) to solve easier (or harder) problems, 2) these RNNs zero-shot generalize to more difficult problem settings not shown during training by dynamically increasing the number of recurrent iterations at test time. Our study provides modeling evidence supporting the hypothesis that recurrent processing enables the functional advantage of adaptively allocating compute resources conditional on input requirements and hence allowing generalization to harder difficulty levels of a visual reasoning problem without training.
1 Introduction
This paper studies whether adaptive recurrent vision models can allocate computation to input difficulty and generalize zero-shot to harder visual reasoning problems. It combines convolutional recurrence with learned halting and evaluates this approach on PathFinder and Mazes.
- Motivation: Adaptive recurrent vision is motivated by humans’ flexible allocation of more computation to harder reasoning problems.Conventional RNNs generally require retraining, fine-tuning, or human intervention to select recurrent iterations across difficulty levels.
- Research aim: The study targets zero-shot extrapolation to harder instances within the same visual reasoning tasks without fine-tuning or human intervention.The evaluation uses the challenging PathFinder and Mazes benchmarks.
- Approach: The authors combine ConvRNNs with Graves’ (2016) adaptive computation method to create AdRNNs that learn both the task and input-dependent computation.The models include recurrent architectures ranging from weight tying to biologically inspired lateral connections and gating.
- Adaptive processing: AdRNNs learn to halt early on easier problems and late on harder problems when training and testing difficulty levels match.This tests whether learned halting reflects per-input computational requirements.
- Zero-shot generalization: At test time, AdRNNs dynamically increase recurrent iterations and zero-shot generalize to harder difficulty levels absent from training.The harder settings are introduced only during evaluation, making computation scaling central to the extrapolation result.
2 Related Work
Prior work established adaptive halting and recurrence-based task extrapolation, but had not sufficiently tested adaptive computation in vision or zero-shot scaling to harder visual problems. This work extends that line with visual reasoning tasks and more specialized recurrent architectures.
- Conceptual connection: Visual routines motivate the use of recurrence because flexible sequencing of elemental operations produces a dynamic computational graph.The paper connects this perspective to visual tasks involving serial grouping and curve tracing.
- Adaptive computation: Graves (2016) developed ACT for NLP RNNs, while Banino et al. (2021) introduced probabilistic halting with a geometric computational-budget prior; neither studied vision RNNs or zero-shot harder difficulty levels.
- Task extrapolation: Earlier recurrence studies generalized from easier to harder problems but required human specification of recurrent steps during testing.They also focused on sequential abstract tasks or straightforward weight-tied recurrent ResNets rather than adaptive visual reasoning.
- Contribution: This work extends task extrapolation to challenging visual reasoning through PathFinder and introduces LocRNN, a recurrent architecture based on cortical recurrence.The authors evaluate both weight-tied recurrence and specialized recurrent architectures for image processing.
3 Datasets
The study evaluates task extrapolation on PathFinder curve tracing and Mazes route segmentation, each constructed with multiple parametric difficulty levels. Training uses easier instances, while larger or longer instances are reserved for zero-shot testing.
- 3 Datasets: The datasets are challenging visual reasoning tasks involving curve tracing or route segmentation across multiple parametric difficulty levels.They are motivated by visual routines and prior psychophysics research on sequential visual processing.
- 3.1 PathFinder challenge – curve tracing: PathFinder classifies whether two disks lie on the same locally connected path among distractor paths.Difficulty is parameterized by path length, with PathFinder-9, PathFinder-14, and PathFinder-18 containing 9, 14, and 18 segments.
- 3.1 PathFinder challenge – curve tracing: PathFinder uses balanced binary classification accuracy, with 700,000 training and 100,000 test images at each standard difficulty level.PathFinder-21 and PathFinder-24 each contain 100,000 images generated exclusively for zero-shot difficulty extrapolation.
- 3.2 Mazes challenge - route segmentation: Mazes is a binary route-segmentation task in which models predict the complete path from a marked start to a marked endpoint.Inputs are RGB maze images containing start, end, permissible, and impermissible regions.
- 3.2 Mazes challenge - route segmentation: Mazes training uses 9×9 and 15×15 grids, while 19×19 and 25×25 grids are reserved for testing and never used in training or hyperparameter optimization.The mixed training set combines the two seen grid sizes.
- 3.2 Mazes challenge - route segmentation: Mazes performance is measured by the percentage of test mazes solved completely accurately at each difficulty level.This criterion gives each maze a single binary score rather than credit for partially correct routes or per-pixel averages.
4 Model architectures and training
The models share convolutional input and readout stages but differ in recurrent blocks, which perform iterative adaptive computation. Adaptive ConvRNNs use ACT to halt based on input processing, while LocRNN adds biologically inspired lateral interactions, gating, and interneurons.
- 4.1 Implementations of adaptive computations evaluated on task extrapolation: Adaptive computation occurs through recurrent convolutional layers applied iteratively for arbitrary timesteps before a convolutional-pooling readout produces the task output.The recurrent block is the model component capable of adaptive computation, with training unrolled up to ttrain timesteps.
- 4.1 Implementations of adaptive computations evaluated on task extrapolation: Three recurrent-block families are evaluated with matched trainable parameters: tied-weight R-ResNet-30, hConvGRU variants and LayerNorm ConvGRU, and the novel biologically inspired LocRNN.Input and readout architectures remain fixed across models, while recurrent cells differ.
- 4.2 Combining ConvRNNs with Adaptive Computation Time (ACT): ACT adds a shared halting convolution that produces a scalar halting score at each recurrent step and stops processing when accumulated scores reach 1 − ϵ.The final adaptive hidden state is a halting-score-weighted average of states up to the stopping timestep, then passed to the readout.
- 4.2 Combining ConvRNNs with Adaptive Computation Time (ACT): Training jointly minimizes task loss and a ponder cost that encourages completing each input with as few recurrent steps as possible.The ponder cost accumulates halting scores until the halting timestep minus one, with τ controlling the objective.
- 4.3 Formulation of LocRNN: LocRNN discretizes an ODE model of primate V1 contour processing, using long-range lateral connections, gating, and separate L and S neural populations.The S population acts as interneurons whose activity is not projected to downstream processing, while recurrent iterations extend interaction range over time.
5 Results
Across PathFinder and Mazes, AdRNNs adapt recurrent computation to input difficulty and extrapolate to harder instances by increasing inference-time iterations. Learnable halting outperforms stability-based halting for extrapolation, while specialized gated recurrent architectures outperform simpler weight-tied recurrence on matched-difficulty tests.
- 5.1 Adaptive RNNs scale their computation as a function of input difficulty: AdRNNs use fewer recurrent steps for easier examples and more steps for harder PathFinder contours and larger mazes, demonstrating input-conditioned computation scaling.The authors report this behavior for both ConvGRU and LocRNN on held-out examples with matched training and test difficulty.
- 5.2 Adaptive RNNs generalize to novel difficulty levels by scaling their computation: AdRNNs extrapolate to harder PathFinder and Maze settings by increasing recurrent iterations at inference, with PathFinder-21 and PathFinder-24 instances using up to 9 LocRNN steps and 7 ConvGRU steps.Maze difficulty, measured by ground-truth route length, is strongly positively correlated with ACT iterations, and longer unseen mazes receive larger halting times than shorter training mazes.
- 5.3 AdRNNs outperform halting in hConvGRU based on stability of hidden-state dynamics: Learnable halting outperforms hidden-state stability halting on both extrapolation datasets, which completely fails to generalize on PathFinder-21 and PathFinder-24, while weight-tied R-ResNet-30 cannot learn PathFinder or Mazes.
6 Conclusion
The study combines adaptive computation with recurrent vision models to scale processing per input difficulty and test zero-shot extrapolation to harder visual reasoning problems.
- 6 Conclusion: AdRNNs trained with ACT dynamically use fewer or more recurrent steps for easy or hard PathFinder and Maze problems, respectively.When test difficulty exceeds training difficulty, they allocate more recurrent computation than was used during training and generalize zero-shot.
- 6 Conclusion: AdRNNs can solve these problems more efficiently by stopping earlier than non-adaptively trained RNNs with human-chosen training iterations.The halting mechanism automatically discovers the number of training iterations required.
7 Limitations and Future Work
The work is limited to static images, while future studies will examine time-varying video inputs and the mechanisms underlying LocRNN’s apparent benefit.
- 7 Limitations and Future Work: The current evaluation covers only static images, so future work will test the models on video input.The authors note that recurrent networks can also process time-varying inputs.
- 7 Limitations and Future Work: The apparent benefit of LocRNN over ConvGRU requires further study to identify the role of its interneuron population.
- 7 Limitations and Future Work: Future work proposes training on very small problem instances while ACT automatically handles harder instances, potentially reducing training and inference latency and energy consumption.
A Training and implementation details
The implementation matches architectures by parameter count, uses task-specific readouts and losses, and trains PathFinder and Mazes under separate schedules.
- A Training and implementation details: All evaluated architectures within each task are matched in trainable parameter count while using task-specific channel widths and recurrent kernel sizes.
- A Training and implementation details: PathFinder uses global-average-pooling classification with binary cross-entropy, whereas Mazes uses a one-channel segmentation map with pixel-wise binary cross-entropy.
- A Training and implementation details: Mazes training uses batches of 64 for 50 epochs with warmup and step decay, while PathFinder uses batches of 256, learning rate 1e-4, and 20 epochs.
- A Training and implementation details: All models were trained on NVIDIA RTX A6000 GPUs and implemented in PyTorch.
B Instability of other baseline ConvRNNs
The baseline comparison examines ConvRNN performance across three PathFinder difficulty levels and highlights instability on harder settings.
- B Instability of other baseline ConvRNNs: LocRNN was the only evaluated model to converge stably across random seeds on difficult PathFinder levels, while other networks performed at chance.
- B Instability of other baseline ConvRNNs: Figure 6 compares various ConvRNN models on PathFinder-9, PathFinder-14, and PathFinder-18.
C Input and output format for PathFinder and Mazes
Mazes require predicting a segmented solution path on an n × n RGB input, whereas PathFinder requires classifying whether two disks are connected through a contour.
- Mazes map n × n RGB inputs with green start and red end squares to binary n × n segmentation masks marking the solution path.Figure 7 (top) illustrates the input and segmented-path output format.
- PathFinder maps n × n binary matrices to two class probabilities indicating whether the disks are disconnected or connected through the contour.The negative class denotes disks at the ends of disconnected paths, while the positive class denotes contour connectivity.
- LocRNN state activations for PathFinder-14 suggest contour integration from both endpoints, followed by a decision based on whether the contours meet.Figure 8 presents this activation-based processing strategy.