PaperlayerView exact revision on arXiv

Source-linked AI summary

On Locality and Length Generalization in Visual Reasoning

Pulkit Madan, Sanjay Haresh, Reza Ebrahimi, Sunny Panchal, Apratim Bhattacharyya, Roland Memisevic

arXiv:2607.09061v1cs.CVcs.AIcs.LG

TL;DR

Global vision models may learn shortcuts that fail on longer visual reasoning problems. This paper tests recurrent models with local perception and finds that combining recurrence and locality enables length generalization.

  • Problem

    The paper asks whether sequential local perception provides computational benefits for visual reasoning and supports length generalization beyond globally encoded images.

  • Method

    The authors evaluate simple visual state-tracking tasks of varying lengths using recurrent vision policies that process foveated and peripheral local glimpses.

  • Results

    Combining recurrence with strictly local perception enables out-of-distribution length generalization, while glimpse size and resolution affect robustness.

  • Takeaways & Limitations

    The results suggest that recurrence and locality are both necessary for compositional generalization, while foveated perception can balance generalization with image search.

  • Takeaways & Limitations

    The local perception policies rely on imitation learning from ground-truth policies, and learning broadly general policies remains an open problem.

Abstract

from arXiv · show

A striking feature of the human visual system is that it ingests visual information through a series of local foveated glimpses, rather than a single global computation. This makes human vision distinctly different from most popular computer vision models in use today, which input images globally and in a single shot. A natural question therefore is whether local, sequential vision models may provide any fundamental computational benefits in addition to being biologically more plausible than global models. In this work, we investigate this question from the perspective of visual state tracking and length generalization. Inspired by recent studies of length generalization in language models, we study the behavior of vision models trained on simple vision tasks that require the aggregation of local information across an image. Our experiments reveal that, similar to language models, vision models can learn to exploit global shortcuts and thereby fail to generalize over task length or complexity. We also show that recurrent vision policies based on strictly local perception can mitigate these failures, thereby allowing models to generalize on these tasks. Our results show that local attention may be an essential overlooked requirement for robust compositional generalization.

1 INTRODUCTION

The paper introduces length generalization as a visual reasoning testbed requiring aggregation of local information across increasingly complex images. It shows that global perception can rely on shortcuts that fail out of distribution, whereas recurrent strictly local perception supports generalization.

  • Testbed: Length generalization tests whether models extrapolate to images with more objects, longer visual dependencies, and greater task complexity than seen during training.The tasks define complexity by the minimum number of steps required to solve them.
  • Failure mode: Current vision and vision-language models can perform strongly in distribution yet fail to extrapolate under this visual reasoning testbed.This exposes a gap not captured by standard benchmark evaluations.
  • Failure mode: Global perception causes failures because processing the entire image at once enables perceptual shortcuts that break as visual complexity increases.These shortcuts fail to represent the tasks’ compositional structure and therefore do not generalize out of distribution.
  • Local processing: Recurrent and strictly local perception enables length generalization, while the size and resolution of local glimpses are crucial for robust generalization.Local sequential processing decomposes reasoning into repeated observations and state updates.
  • Implication: Across three synthetic tasks and one real-world task, the results suggest flexible, human-like reasoning may require biologically inspired sequential attention policies.The work emphasizes that models must gather local information as well as follow an abstract reasoning process.

2 TEST-BED FOR LENGTH GENERALIZATION

The test bed evaluates visual reasoning models on systems whose states are encoded by local image information, varying resolution and task length. It includes state-tracking tasks, a recall control, and mathematical plot reasoning under in-distribution and out-of-distribution settings.

  • Task dimensions: The tasks vary generalization along image resolution and task length, with task length defined by the number of local information pieces encoding the system state.Resolution and task length are distinct dimensions but can interact because entity density depends on resolution.
  • State-tracking tasks: Visual Parity requires aggregating spatially distributed binary switches to determine whether the number of pressed states is odd or even.Unlike textual parity, the switches are not presented in a pre-ordered sequence; the model must locate them in the 2D image.
  • State-tracking tasks: State Machine tests state tracking when aggregation order matters, complementing Visual Parity’s permutation-invariant aggregation.The task simulates a state machine realizing an action of the dihedral group.
  • Control task: Recall provides a control condition where models search for a target object among distractors, allowing out-of-distribution generalization without state tracking.The example target is a red letter “L” among red “T” and green “L” distractors.
  • Real-world reasoning task: Finding Roots evaluates mathematical reasoning over plots by requiring iterative localization of subplots, functions, axes, and curve intersections with y = 0.Out-of-distribution scenarios vary the number of zeros and/or subplots relative to training.
  • Dataset construction: Datasets are generated from oracle trajectories containing local and peripheral glimpses, actions, and probes, with training restricted to in-distribution data and evaluation on both settings.Unless specified otherwise, the in-distribution resolution is 1200 × 800.

3 RECURRENT FOVEATION MODEL

FOVEAGENT-LSTM is a recurrent vision model that processes images through sequential foveated and peripheral glimpses, using local information to choose subsequent locations and solve tasks. Its architecture combines ResNet encoders for both glimpse types with an LSTM backbone and can provide auxiliary probe outputs during training.

  • Overview and Architecture: FOVEAGENT-LSTM ingests visual input through a sequence of foveated and peripheral glimpses, rather than a single global computation.The model is designed as a recurrent, local vision model and is illustrated in Fig. 3.
  • Overview and Step-by-step Operation: Starting from a fixed initial location, the model iteratively processes local glimpses, predicts displacements for subsequent locations, and stops after acquiring task-relevant information.At each iteration, it generates the next glimpse location together with any additional task-specific information.
  • Architecture: The architecture uses separate ResNet encoders for foveated and peripheral glimpses, concatenates their features, and feeds them into a recurrent LSTM backbone.The peripheral glimpse is 4 × the size of the foveated glimpse, while both are resized to the same sensor resolution.
  • Architecture: At each timestep, the model predicts the next glimpse displacement as an angle–magnitude action, with angle θ in [0, 2π] and magnitude d in [0, Gp].The next location is defined by xt+1 = xt + d u_t(θ).
  • Probes: The model can emit probes alongside actions, providing supervision beyond final-answer accuracy; in VISUAL PARITY and STATE MACHINE, probes indicate whether the current local state is 0, 1, or not visible.Probe values are 0, 1, or null, respectively.

4 EXPERIMENTS

Experiments show that foveated local perception with recurrent processing generalizes across task length, complexity, and resolution, whereas global or non-recurrent models degrade out-of-distribution. The RECALL result further indicates that global models can excel when state tracking is unnecessary.

  • Experimental Setup: Training uses 2–10 switches at 800x1200 resolution, while OOD evaluation varies resolution and includes 11–20 switches.FOVEAGENT-LSTM is compared with QWEN2.5-VL-3B-INSTRUCT and other closed-source VLMs.
  • VISUAL PARITY and STATE MACHINE: FOVEAGENT-LSTM generalizes across task complexity, resolution, and task length, while QWEN2.5-VL-3B-INSTRUCT declines sharply as complexity increases.QWEN2.5-VL-3B-INSTRUCT achieves high in-distribution accuracy but fails to maintain it OOD.
  • Visual Interface: Global and Local+Global interfaces show poor OOD generalization despite similar in-distribution performance, whereas foveated local glimpses maintain high success.The result supports the necessity of local perception for length generalization.
  • Recurrence: Recurrent networks are the only models that length-generalize, indicating that non-recurrent models can learn shortcuts even with local visual input.The experiment provides evidence for H2, the necessity of recurrence for OOD generalization.
  • RECALL: QWEN2.5-VL-3B-INSTRUCT significantly outperforms the foveated model on RECALL, where global vision can identify binding features without state tracking.This contrast shows that the benefits of foveated processing are tied to tasks requiring visual state tracking.

5 RELATED WORK

Prior work documents out-of-distribution length failures in language and vision, while recurrent and biologically inspired attention models motivate this study’s focus on local perception for visual generalization.

  • OOD Generalization in Language Tasks: Language models often fail algorithmic tasks when token-sequence lengths differ from training lengths, with proposed explanations involving globality and random-access addressing.The passage introduces several explanations but is truncated before detailing the random-access account.
  • OOD Generalization in Vision Tasks: Vision research includes broad visual-reasoning and out-of-distribution benchmarks, whereas this work studies generalization over the number of visual entities.The passage characterizes the paper’s setting as a highly specific and simple form of visual OOD generalization.
  • OOD Generalization in Vision Tasks: Veerabadran et al. (2023) use recurrent networks for visual OOD generalization, but their models perceive whole images rather than local regions.Their recurrent models outperform non-recurrent ones, yet generalization drops sharply with image size, consistent with this paper’s findings.
  • Visual attention: Biologically inspired spatial attention has been studied in computer vision, including foveation-based approaches, but recent work has favored convolutional or ViT-based vision-language front-ends.The passage frames spatial attention as having taken a back-seat in recent years.

6 DISCUSSION · A APPENDIX OVERVIEW

The discussion argues that combining local perception with recurrent processing enables visual length and compositional generalization, while foveated perception resolves exploration–generalization tradeoffs. It also identifies policy-learning limitations, future hypotheses, and the Appendix’s supporting materials.

  • 6 DISCUSSION: Local perception combined with recurrent processing unlocks length generalization and may be necessary and sufficient for compositional generalization in visual reasoning.The authors hypothesize that global context windows or global vision front-ends prevent out-of-distribution generalization.
  • 6 DISCUSSION: Out-of-distribution generalization favors small, low-resolution glimpses, whereas image search favors large, high-resolution glimpses.A high-resolution fovea with a low-resolution periphery resolves this conflict.
  • 6 DISCUSSION: The local perception policies rely on imitation learning from ground-truth policies, limiting immediate generality beyond synthetic tasks.Broader general policies may require more varied tasks, reinforcement learning, or both.
  • 6 DISCUSSION: The authors hypothesize that recurrent computation could improve visual reasoning data efficiency and that local visual attention may be crucial for future visual foundation models.This hypothesis is motivated by reported benefits of recurrence for language-model reasoning and out-of-distribution generalization.
  • A APPENDIX OVERVIEW: The Appendix begins with evaluation metrics for each task and dataset and task-construction details, including the formal STATE MACHINE construction.These materials are provided in Sections B and C.
  • A APPENDIX OVERVIEW: Sections D and E provide implementation, training, and additional controlled visual-reasoning results supporting analyses of local versus global perception and recurrent versus global processing.Section D covers FOVEAGENT-LSTM and FOVEAGENT-QWEN; Section E covers the controlled results.
  • A APPENDIX OVERVIEW: Section F reports additional FINDING ROOTS results.This is the final Appendix item listed in the provided overview passage.

B EVALUATION METRICS

Models are evaluated by final-state prediction accuracy, while FOVEAGENT-LSTM additionally must attend to every task-relevant canvas object. Missed items are penalized under this traversal requirement.

  • Evaluation criteria: Visual-Parity and State-Machine use prediction accuracy to assess whether models identify the correct final state.Targets include parity states 0/1 and state-machine states 0/1/2.
  • Evaluation criteria: FOVEAGENT-LSTM must attend to every task-relevant canvas object, and missing any item is penalized.This traversal requirement applies in addition to identifying the final state.
  • Evaluation criteria: Relaxed evaluation compares the set of model-predicted zeros, x, with the ground-truth zero set, Z.The passage defines |x| as the number of zeros predicted by the model.

C DATASET AND TASK DETAILS

The paper constructs in-distribution and out-of-distribution visual reasoning datasets by varying image resolution and task complexity, training only on in-distribution examples. Its STATE MACHINE task requires order-sensitive processing of visually presented switches to determine a final state position.

  • Dataset and split construction: Each task uses separate training, validation, in-distribution test, and additional out-of-distribution test sets, with models trained only on in-distribution examples.The splits vary image resolution and task complexity.
  • Trajectory construction: Glipses are down-sampled to sensor resolution, and positional noise is added to oracle trajectories so policies learn recovery behavior.Except for FINDING ROOTS, training uses 50k trajectories at 1200 × 800 resolution.
  • STATE MACHINE construction: STATE MACHINE is order-sensitive: models must process switch values in arrow-specified order rather than merely aggregating the observed set.The task instantiates a visual sequence implementing the action of the dihedral group.
  • STATE MACHINE construction: The semiautomaton state is (s, b), where advance moves one step around a cycle and reverse flips the direction; experiments use S = 3.The transition rules are δ((s, b), advance) = (s + b (mod S), b) and δ((s, b), reverse) = (s, −b).
  • STATE MACHINE construction: Given an initial state from the prompt, the task processes the full switch sequence and outputs the final position sn ∈ {0, 1, 2}.The input sequence is visually represented by switches connected with arrows.

D IMPLEMENTATION DETAILS

FOVEAGENT-LSTM uses a recurrent visual architecture with ResNet-18 and local-plus-peripheral glimpses, while FOVEAGENT-QWEN builds on QWEN2.5-VL-3B-INSTRUCT for the Finding Zeros task. The QWEN baseline receives a global image and simulates an imaginary glimpse, whereas FOVEAGENT-QWEN supplements this trajectory with local glimpses.

  • FOVEAGENT-LSTM: FOVEAGENT-LSTM uses a 4-layer LSTM as its recurrent model and ResNet-18 as its visual backbone.The implementation is explicitly described as the base recurrent and visual architecture.
  • FOVEAGENT-LSTM: Each FOVEAGENT-LSTM step receives two glimpses: an 80 × 80 local glimpse and a peripheral glimpse parameterized by x.The default setting is x = 4, with x ∈ {1, 2, 4, 8, 16} unless otherwise specified.
  • FOVEAGENT-QWEN: FOVEAGENT-QWEN uses QWEN2.5-VL-3B-INSTRUCT as its foundation model and fine-tunes the base model on Finding Zeros as a baseline.The baseline receives a full-resolution global image of 1200 × 800 and simulates an imaginary 200 × 200 glimpse to locate function roots.
  • FOVEAGENT-QWEN: FOVEAGENT-QWEN follows the baseline’s trajectory while supplementing the global glimpse with local glimpses.This contrasts the baseline’s global-image procedure with the proposed model’s additional local visual inputs.

E ADDITIONAL CONTROLLED VISUAL REASONING RESULTS · E.1 1D VISUAL PARITY: CONTROLLING FOR THE VISUAL FRONT-END

The controlled results separate visual evidence gathering from observation processing and test whether visual parity models generalize when sequences extend beyond training lengths. A local recurrent model generalizes substantially farther than a global model, which performs well in-distribution but fails on longer out-of-distribution sequences.

  • E ADDITIONAL CONTROLLED VISUAL REASONING RESULTS: The paper separates visual evidence gathering from processing the resulting observations as two ingredients of out-of-distribution visual reasoning.The Local vs Global Perception experiment studies the visual interface, while Recurrent vs Global Processing studies computation over local visual inputs.
  • E.1 1D VISUAL PARITY: CONTROLLING FOR THE VISUAL FRONT-END: The 1D visual parity task preserves visual input while making the sequence structure explicit.Bit sequences are rendered as horizontal images, with white and gray blocks representing 0s and 1s, respectively.
  • E.1 1D VISUAL PARITY: CONTROLLING FOR THE VISUAL FRONT-END: The 1D task controls for the visual front-end difference between 2D visual parity and standard language parity.Unlike language parity, the visual task requires extracting relevant symbols through a visual front-end.
  • E.1 1D VISUAL PARITY: CONTROLLING FOR THE VISUAL FRONT-END: With symbols presented sequentially, the central test is whether models repeatedly apply the same parity update beyond the training length.This setting directly probes processing of local visual inputs.
  • E.1 1D VISUAL PARITY: CONTROLLING FOR THE VISUAL FRONT-END: The local recurrent model generalizes to substantially longer sequences than those seen during training.This result is reported for the 1D visual parity setting in Fig. 11.
  • E.1 1D VISUAL PARITY: CONTROLLING FOR THE VISUAL FRONT-END: The global model achieves strong in-distribution performance but fails to generalize to longer out-of-distribution sequences.The result mirrors the gap between recurrent networks and transformer-style models on language parity.
  • E.1 1D VISUAL PARITY: CONTROLLING FOR THE VISUAL FRONT-END: The 1D visual parity findings support the conclusion that recurrence is important for robust length generalization.The conclusion follows from the contrast between local recurrent and global models on longer sequences.

E.2 ADDITIONAL VISUAL PARITY BASELINES

Additional VISUAL PARITY baselines reinforce that strict recurrent processing supports stronger out-of-distribution generalization as task length increases, whereas global or non-strict processing is more vulnerable to shortcut solutions. Together with the main experiments, the results indicate that robust visual state tracking requires both appropriate perception and recurrent state updates.

  • Local vs Global Perception: With full high-resolution visual input, Global and Local+Global variants perform well in distribution but degrade OOD, unlike FOVEAGENT-LSTM.FOVEAGENT-LSTM combines local high-resolution glimpses with low-resolution peripheral context.
  • Recurrent vs Global Processing of Local Visual Inputs: When the visual input stream is fixed, strict recurrent state updates generalize OOD better than attention-based transformers, Mamba, and xLSTM.The richer-global-interaction architectures degrade more sharply over the observation history.
  • VISUAL PARITY: Strict recurrent backbones maintain stronger OOD generalization on VISUAL PARITY as task length increases.This matches the reported pattern in the additional baselines and the main-paper STATE MACHINE results.
  • VISUAL PARITY: Non-strictly-recurrent or globally processing backbones can fit training-length regimes but are more prone to shortcut failures as task length grows.The figure characterizes this degradation as increasing numbers of switches are added.

F ADDITIONAL RESULTS ON FINDING ROOTS · G ZERO-SHOT EVALUATION PROMPTS

The additional FINDING ROOTS setting requires identifying a target function among multiple functions and estimating its roots, highlighting the role of local visual inspection. The zero-shot evaluations specify structured procedures for parity, state-machine, and object-presence tasks, with reasoning in <think> and answers in <ans>.

  • F ADDITIONAL RESULTS ON FINDING ROOTS: The multi-function FINDING ROOTS split requires finer visual processing to identify the target function and estimate its roots.The target subplot contains two or more functions, making local glimpses especially beneficial.
  • F ADDITIONAL RESULTS ON FINDING ROOTS: The FINDING ROOTS evaluation compares a global-view QWEN2.5-VL-3B-INSTRUCT baseline with FOVEAGENT-QWEN variants that add high-resolution local glimpses.The baseline receives only the full-image input G, whereas FOVEAGENT-QWEN combines G with local glimpses L.
  • F ADDITIONAL RESULTS ON FINDING ROOTS: FINDING ROOTS requires locating the target subplot and legend, scanning along y = 0, and interpolating x-values where roots are needed.The oracle trajectory first localizes the subplot and legend, then searches for function intersections with the zero level.
  • G.1 VISUAL PARITY PROMPT: Visual Parity prompts instruct models to find all circular nodes, read their 0-or-1 digits, form a bit string, and compute parity.Parity is 1 for an odd number of 1s and 0 for an even number; the final value is returned inside <ans> tags.
  • G.2 STATE MACHINE PROMPT: State Machine prompts require following arrows from the green start through beige nodes to the purple end while updating a modulo-3 state.A 0 applies the current increment/decrement operation, whereas a 1 reverses that operation without directly changing the numerical state.
  • G.3 RECALL PROMPT: Recall prompts ask models to inspect the canvas step by step, determine whether a specified colored object is present, and return True or False.The required final response is placed inside <ans> tags after visual reasoning in <think> tags.
Loading 2607.09061v1…