Source-linked AI summary
Learning long-range spatial dependencies with horizontal gated-recurrent units
Drew Linsley, Junkyung Kim, Vijay Veerabadran, Thomas Serre
TL;DR
CNNs struggle with recognition tasks requiring co-dependent visual features over long spatial ranges, despite strong performance on many visual tasks. The paper introduces the horizontal gated-recurrent unit (hGRU), which learns horizontal spatial interactions through recurrent processing. A single hGRU layer matches or outperforms tested feedforward baselines with far fewer parameters, while showing similarities to visual-cortex connectivity and human contour-detection behavior.
Problem
CNNs struggle to detect co-dependent visual features over long spatial ranges, and simply increasing feedforward depth is an inefficient solution to this dependency.
Method
The hGRU is a trainable recurrent module that learns horizontal interactions through gated hidden-state updates and spatial kernel W.
Results
A single hGRU layer reliably detects Pathfinder paths of tested lengths and forms, matching similar success from much deeper feedforward architectures with orders of magnitude more parameters.
Takeaways & Limitations
The hGRU provides a biologically inspired solution for representing extended paths while suppressing surrounding clutter, connecting machine learning with visual neuroscience and cognition.
Takeaways & Limitations
The hGRU relaxes several constraints from the original neuroscience model because they were less useful for the investigated tasks.
Abstract
from arXiv · showhide
Progress in deep learning has spawned great successes in many engineering applications. As a prime example, convolutional neural networks, a type of feedforward neural networks, are now approaching -- and sometimes even surpassing -- human accuracy on a variety of visual recognition tasks. Here, however, we show that these neural networks and their recent extensions struggle in recognition tasks where co-dependent visual features must be detected over long spatial ranges. We introduce the horizontal gated-recurrent unit (hGRU) to learn intrinsic horizontal connections -- both within and across feature columns. We demonstrate that a single hGRU layer matches or outperforms all tested feedforward hierarchical baselines including state-of-the-art architectures which have orders of magnitude more free parameters. We further discuss the biological plausibility of the hGRU in comparison to anatomical data from the visual cortex as well as human behavioral data on a classic contour detection task.
1 Introduction
CNNs perform well on natural-scene contour detection but struggle to integrate long-range spatial dependencies in Pathfinder-like tasks. The paper motivates horizontal connections as an alternative to simply increasing feedforward depth and introduces the hGRU as a trainable solution.
- Motivation: Pathfinder tests whether two circles are connected by a potentially curved path amid shorter unconnected distractor paths.Path length and distractor complexity are parametrically increased across three datasets.
- Motivation: CNNs’ localized receptive fields limit their ability to learn long-range spatial dependencies, making greater depth an inefficient solution.The paper contrasts this limitation with CNN success on natural-scene contours.
- Biological motivation: Visual-cortex horizontal connections provide a biological alternative by enabling non-linear interactions across distal visual-field regions.These association fields are linked to contour grouping through excitation of colinear elements and suppression of clutter.
- Contribution: The hGRU converts a computational neuroscience model into an end-to-end trainable module for learning intrinsic spatial interactions.It is designed to address long-range dependencies without relying on deep feedforward stacks.
- Contribution: The hGRU solves Pathfinder effectively with one layer and fewer parameters and training samples than CNNs, while learning connection patterns resembling visual-cortex anatomy.Its natural-scene detection profile also strongly correlates with human behavior on a classic contour task.
2 Horizontal gated recurrent units (hGRUs)
The hGRU is a trainable convolutional recurrent module that uses gated, recurrent horizontal interactions to transform feedforward activity into contextual hidden states. Its design combines inhibition, excitation, symmetric spatial kernels, and biologically motivated recurrent dynamics within a single processing layer.
- Core formulation: A trainable convolutional recurrent network results by applying Euler integration to a dynamical system modeled on a recurrent neural circuit.Discrete-time states approximate the continuous-time dynamics at each timestep.
- Core formulation: The hGRU adds learnable gates and symmetric excitation operations to improve gradient-based training and expressive power.These modifications extend the original contextual circuit model.
- Core formulation: The hGRU receives feedforward activity X and encodes spatial dependencies through time-varying hidden states H(1) and H(2).H(1) is the recurrent circuit input and H(2) the recurrent circuit output.
- Horizontal interactions: The kernel W computes horizontal interactions across spatial positions and feature channels, with symmetric channel weights nearly halving parameters relative to a normal convolution.Its spatial extent S determines the range of interactions within one timestep.
- Inhibition: The first stage computes gated horizontal inhibition from H(2)[t−1], applies it to X through linear and quadratic terms, and updates H(1)[t].The hGRU uses H(2)[t−1], rather than H(1)[t−1], in this update because this improved learning on the evaluated visual tasks.
- Excitation: The second stage computes horizontal excitation from H(1)[t], modulates it with linear and quadratic terms, and mixes it with the persistent H(2)[t−1] state.A peephole connection from H(1)[t] contributes to the excitation computation.
3 The Pathfinder challenge
The Pathfinder challenge tests whether models can integrate increasingly long visual paths amid distractors. Across controlled feedforward, recurrent, residual, and per-pixel architectures, the hGRU learned these dependencies effectively and with far fewer parameters than most strong baselines.
- Task and evaluation: Pathfinder images require detecting whether two circles are connected by a curved path while distractor paths increase task difficulty.The three datasets vary path length, with paths containing 6, 9, or 14 paddles.
- Task and evaluation: The evaluation replaced the feature-extraction stage of a standard input–feature extraction–readout architecture with different feedforward or recurrent layers.This controlled model comparison included six recurrent layers and 15 feedforward configurations.
- Feedforward and recurrent performance: Increasing path length reduced accuracy and increased the number of batches required for learning across models.This strain was measured through accuracy and area under the learning curve (ALC).
- Feedforward and recurrent performance: The 8-timestep hGRU outperformed the other recurrent models and showed only a negligible accuracy drop as path length increased.It also outperformed a standard GRU and hGRU variants with lesions to linear or quadratic horizontal interactions.
- Feedforward and recurrent performance: Feedforward models needed greater depth to handle longer paths, but even five-layer models largely failed on the 14-length dataset.The large-kernel five-layer model approached solving that condition but learned more slowly than the hGRU despite having 1×–10× its parameters.
- State-of-the-art baselines: The hGRU matched or exceeded strong per-pixel and residual baselines while using far fewer parameters.The FCN used 200×, U-Net over 350×, SegNet 400×, and the 152-layer ResNet nearly 1000× as many parameters as the hGRU.
4 Explaining biological horizontal connections with the hGRU
The authors examine whether hGRU connectivity resembles biological association fields and whether its behavior relates to human contour-detection performance. Learned horizontal kernels showed structured patterns in natural-scene training, providing the basis for this biological comparison.
- Biological connectivity: The hGRU was analyzed for horizontal connection patterns resembling cortical association fields, whose geometry reflects oriented elements in natural scenes.The analysis asked whether the model learned connectivity patterns similar to those suggested by cortical image statistics.
- Biological connectivity: Training on the naturalistic BSDS500 contour dataset produced more defined and regular horizontal connectivity patterns.The patterns were similar to those observed in the broader analysis but showed greater definition and regularity after natural-scene training.
- Human psychophysics: The authors compared hGRU contour-detection accuracy with human participants across five conditions varying the distance between contour paddles.The task used co-linearly aligned paddles among randomly oriented distractors.
5 Discussion
The study identifies long-range spatial dependencies as a weakness of feedforward networks and presents the hGRU as a biologically inspired alternative. The hGRU solves tested Pathfinder conditions with one layer while learning connection patterns and behavioral tradeoffs resembling visual-cortex and human data.
- Long-range spatial dependencies generally strain CNNs, with only very deep and state-of-the-art networks overcoming variability introduced by long paths.
- The hGRU reliably detects paths of any tested length or form using one layer, whereas successful feedforward alternatives use much deeper architectures and orders of magnitude more parameters.
- hGRU processing time affects Pathfinder performance, consistent with iterative computation that facilitates target-curve influence while suppressing distractors.
- Horizontal kernels learned from natural scenes resemble cortical association fields and paired near-excitatory/far-inhibitory surrounds.
- The hGRU’s weights and behavior appear consistent with visual-cortex patterns and may connect machine learning, cognitive science, and neuroscience.
Deriving the hGRU from the contextual neural circuit model
The hGRU is derived by discretizing a contextual neural circuit model into a convolutional recurrent system and then adding learned gates. Its design modifies standard GRU gain and mixing operations to support the separate horizontal-interaction stages needed for Pathfinder.
- The continuous contextual model is discretized with Euler’s method into a difference equation that can be viewed as a convolutional RNN with ReLU nonlinearity.
- Learned gates manage information flow over time by integrating past information with new computations more flexibly than the Euler approximation.
- The hGRU uses convolution kernels U and bias vectors b within gated horizontal computations, with σ as a pointwise squashing nonlinearity.
- Unlike the reference GRU, the hGRU’s mix depends on H(1)[t] rather than H(2)[t −1], and its separate stages are critical for Pathfinder.
- The hGRU applies GRU-style gain to the first horizontal-interaction stage and uses a learned per-timestep η instead of ϵ^-2 scaling.
The Pathfinder challenge
The Pathfinder challenge tests whether a system can determine connectivity between distant target circles amid curved paths and distractors. Its generator varies path geometry while enforcing spacing and non-contact constraints.
- The task uses 150×150 binary images containing two white circles, two target paths, and multiple distractor paths on black backgrounds.
- Each path is built from k co-circularly arranged paddles, with seed and trailing paddles added inductively.
- Target-path seeds are positioned at intersections of a randomly oriented line and an invisible circle, keeping the two targets sufficiently close.
- Trailing-paddle orientations are sampled from angle differences, while continuity c controls path rigidity and margin m controls spacing between adjacent paddles.
- Distractor paths are independently seeded and fill the image to 150 total paddles, preventing shortcuts based on loose ends or isolated circles.
- New paddles are rejected and resampled when they contact another paddle or violate the minimum separation m.
The Synthetic Contours Dataset
The Synthetic Contours Dataset provides highly variable contour-detection stimuli for comparing an hGRU with human behavioral data. It parametrically varies contour length and relative spacing while controlling contour geometry and distractors.
- SCD was created to compare an hGRU pretrained on BSDS500 with human contour-detection behavior.
- The dataset contains 28 million images with practically infinite variability in contour position, orientation, and relative spacing.
- All SCD images use 256×256 pixels and span a radius of 4 visual degrees.
- Each contour is generated by filling l neighboring grid cells along a grid diagonal, with paddles connecting diagonally opposite points to form a collinear contour.
- SCD varies contour length and relative spacing, using contours of 5, 9, 14, or 17 paddles.
Model architecture details
The study compares recurrent and feedforward feature-extraction stages within a standardized architecture, while also testing hGRU variants and parameter-constrained baselines. The hGRU uses recurrent horizontal processing, whereas the control models vary normalization, recurrence depth, feedforward design, or parameter count.
- Comparison framework: All recurrent and feedforward models share the same input and readout stages, enabling direct comparison of their feature-extraction architectures on Pathfinder.The shared pipeline separates preprocessing, feature extraction, and classification.
- Input and readout: The input stage uses 25 7×7 convolutional kernels initialized as oriented Gabor filters and a radially symmetric difference-of-Gaussian filter.Filter activity is squared pointwise before entering the model-specific feature-extraction stage.
- hGRU variants: The hGRU-BN adds batch normalization to multiple circuit computations and remains effective on Pathfinder while improving training stability for nonsquashing nonlinearities.This enabled a rectified-linear hGRU variant with constrained inhibition and excitation roles.
- Control models: A two-layer GRU improved performance on the 6-length Pathfinder dataset but failed on the longer 9- and 14-length versions.Thus, adding a second recurrent processing layer did not reproduce the hGRU’s performance across dataset lengths.
- Feedforward controls: Highway-network models performed similarly to the large-kernel feedforward configuration in the additional architecture screening.The screening also included one-, three-, and five-layer highway configurations.
- Parameter-constrained baselines: Constrained ResNets solved the 6- and 9-length datasets but failed on the 14-length dataset, whereas constrained U-Nets performed better and only the most constrained version was strained there.The U-Net comparison still required over an order of magnitude more free parameters than the hGRU.
Training on natural images and comparisons to human data
The hGRU learns horizontal connectivity patterns from Pathfinder and natural-image contour detection that resemble canonical visual-cortex organization. After natural-image training, its connectivity is cleaner, and its performance under increasing contour gaps shows strain similar to human observers.
- Connectivity patterns: hGRU horizontal kernels learned from Pathfinder and natural scenes resemble cortical association fields and paired near-excitatory and far-inhibitory surrounds.These patterns reflect oriented-feature connectivity reported in theoretical models of visual cortex.
- Connectivity analysis: The connectivity analysis aligns kernels to a common reference orientation and uses PCA to identify and denoise shared patterns.This procedure produces the eigenconnectivity visualizations used to compare learned patterns across tasks.
- Pathfinder connectivity: Pathfinder eigenconnectivity includes antagonistic near-excitatory versus far-inhibitory surrounds, collinear excitation with orthogonal inhibition, and higher-order surround computations.These are described as canonical patterns of cortical connectivity.
- Natural-image training: Horizontal-kernel eigenconnectivities are cleaner and more regular after training on natural-image contour detection than after Pathfinder training.The authors relate this difference to the image statistics useful for path integration versus contour detection.
- Experimental design: For the human-comparison experiment, 1,000,000 unique 150×150 images were generated per condition, with 900,000 for training and 100,000 for testing.The BSDS-trained hGRU was fine-tuned separately on each condition before evaluation.
- Comparison with human data: The hGRU’s accuracy and human detection scores both become strained as the distance between contour paddles increases.The hGRU was overall more accurate than humans, but both showed a similar distance-related performance pattern.