PaperlayerView exact revision on arXiv

Source-linked AI summary

LoopVL: Recurrent Visual Intelligence

Zhe Qian, Ziyang Gong, Zhongxing Xu, Hehan Li, Zhonghua Wang, Fei Luo, Mingxuan Wang, Xue Yang, Shiwei liu, Yanbiao Ma, Junchi Yan, Jungong Han

arXiv:2609.38426v1cs.CVcs.CL

TL;DR

Vision–language recurrence must operate on states that evolve as instructions change which spatially organized visual evidence matters. LoopVL applies shared recurrent computation to a unified vision–language state and evaluates it across multimodal tasks, finding recurrent visual evidence reorganization while noting that broader architectural controls remain untested.

  • Problem

    Recurrent studies have largely focused on language, leaving the question of whether they extend effectively to vision–language models whose relevant visual regions may shift with the instruction.

    Source: §1 · p. 3

  • Method

    LoopVL repeatedly updates a unified vision–language state through shared recurrent computation, and the paper analyzes attention and visual-state changes across recurrent steps.

    Source: §1 · p. 3

  • Results

    LoopVL is evaluated across multimodal tasks and recurrent configurations, and later recurrent steps reorganize and reuse visual evidence rather than merely repeating earlier computation.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Spatially corresponding visual tokens provide an interpretable way to examine how shared modules process visual evidence again across recurrent steps.

    Source: §1 · p. 3, §8 · p. 24

    Without a matched control using a more general Model-Loop architecture, the observed Visual Aha Moments are not established as intrinsic to Model-Loop computation.

    Source: §7 · p. 23

Abstract

from arXiv · show

We introduce LoopVL to study whether Loop Transformers can be effectively extended to vision- language models. LoopVL combines Module-Loop and Model-Loop computation to iteratively update a unified vision-language state through shared modules. We train LoopVL from scratch through language pre-training, multimodal training, and post-training. LoopVL outperforms a range of similarly sized and larger non-recurrent models on multimodal understanding and visual reasoning benchmarks. We also observe Visual Aha Moments in LoopVL, characterized by pronounced shifts in visual attention across loops. LoopVL provides practical evidence for recurrent vision-language modeling and offers an intuitive perspective on how shared parameters can support deeper multimodal computation over continuously evolving visual-language states.

1 Introduction

The paper asks whether recurrent computation can extend effectively to vision–language models, where visual evidence and instruction relevance may evolve across steps. LoopVL applies shared recurrent computation to a unified multimodal state and studies how visual evidence changes across those steps.

1 Introduction

  • Existing recurrent-model studies largely focus on language, while vision–language recurrence must update visual and language states as query-relevant regions may change.The paper highlights whether later steps continue attending to the same regions or reallocate attention to previously overlooked ones.Source: §1 · p. 3
  • Spatial correspondence between visual tokens and image patches makes recurrent state transitions observable as changes in attention to concrete image regions.This permits comparing the same physical layers across recurrent steps.Source: §1 · p. 3
  • LoopVL repeatedly updates a unified vision–language state with shared parameters, extending recurrent computation to multimodal modeling without adding parameters.Its recurrent computation lets visual information and language instructions interact across multiple stages.Source: §1 · p. 3
  • The work evaluates vision–language pretraining and post-training across multimodal tasks and recurrent configurations as empirical evidence for multimodal recurrence.The architecture combines Module-Loop and Model-Loop computation over the visual-language state.Source: §1 · p. 3, Figure 2 · p. 4
  • Later recurrent steps reorganize and reuse visual evidence rather than merely repeating earlier computation.The paper studies visual information throughout recurrent computation.Source: §1 · p. 3

2 The LoopVL Model

LoopVL combines nested Module-Loop and Model-Loop computation to update a coupled visual–language state through shared modules. The section motivates this design, compares it with dense Transformers, and examines recurrent visual-state updates.

2.1 Architecture

  • Each L and H stack has 16 Transformer layers with separate parameters reused across invocations; the default execution stores two stacks but performs 128 layer calls.The coupled states initialize differently: H from scaled input embeddings and L to zero, with L retained across model cycles; the final H state feeds the language-modeling head.Source: §2.1 · pp. 4–5

2.2 Motivation

  • The nested L/H design repeatedly refines local visual–language representations under a relatively stable high-level state, then integrates them into a state for the next cycle.Module-Loop supplies denser updates under the same high-level condition, while Model-Loop lets the integrated state re-enter computation.Source: §2.2 · p. 5

2.3 Why Need Loop?

  • LoopVL outperforms the same-unique-layer-count Transformer-VL 1B baseline across evaluated benchmarks, while increasing executed depth from 32 to 128 layer calls.MMStar rises from 55.33 to 63.47, RealWorldQA from 55.29 to 70.98, and ChartQA from 51.12 to 74.52; Table 1 compares models at 0.14T training tokens.Source: §2.3 · p. 6, Table 1 · p. 6
  • With an approximately 1B-scale language backbone, LoopVL approaches or surpasses substantially larger dense Transformers and remains competitive across multimodal tasks.Estimated training FLOPs are 2.47 × 10^21 for LoopVL, versus 2.89 × 10^21 and 2.99 × 10^21 for the two larger models; shared parameters increase executed depth without storing substantially more independent parameters.Source: §2.3 · p. 6

2.4 Why Loop Visual Tokens?

  • Visual tokens continue evolving inside the recurrent backbone without another visual-encoder pass, so shared modules process changed states and can reallocate visual evidence.Spatial correspondence to image patches allows comparing the same physical layer across cycles.Source: §2.4 · p. 6
  • Normal LoopVL achieves the highest accuracy on LogicVista, RealWorldQA, VMCBench-DEV, and MMStar among the visual-update controls.The controls either discard second-cycle L-module visual updates, hold visual states fixed during that cycle, or train a read-only variant that updates them only in the first cycle.Source: §2.4 · p. 6, Figure 3 · p. 7

3 Pre-Training

LoopVL is pretrained as a language model and then adapted for multimodal use through visual alignment and multimodal mid-training. The stages differ in which components are frozen or trainable and in their data and optimization setup.

3 Pre-Training

  • Text pretraining uses 75B tokens: 60B from the HRM-Text base recipe and 15B from additional curated data.The language model is built on HRM-Text’s open-source framework and publicly released data recipe.Source: §3 · p. 7

3.1 Initialization

  • Initialization combines a pretrained Penguin Vision Encoder with the H2L3 language checkpoint; the dynamic-resolution Stage 1 projector is randomly initialized.The visual interface normalizes and calibrates visual features to the median RMS of language embeddings before they enter the backbone path.Source: §3.1 · p. 8

3.2 Learning Schedule

  • The two main stages use global batch sizes of 256 and 128, respectively, with warmup, gradient clipping, and stage-specific learning rates.The main recipes use AdamW, clip gradient norms at 1.0, and warm up for 3% of training steps; Stage 2 uses cosine decay after warmup.Source: §3.2 · p. 8

3.3 Stage 1: Visual Language Alignment

  • Stage 1 trains only the visual interface on LLaVA-559K alignment data while freezing the language backbone and vision encoder.It uses a 0.33B-token allocation, image–caption supervision, and a maximum sequence length of 2048; checks test whether the interface transmits image-dependent information.Source: §3.3 · p. 8

3.4 Stage 2: Multimodal Mid-Training

  • Stage 2 uses OneVision-1.5 Mid-Training-85M, corresponding to the full 49B-token allocation, and jointly trains the language backbone and visual interface for instruction-conditioned generation on aligned visual inputs.Tasks span general visual questions, text-rich images, diagrams, mathematics, and short answers. Over-budget responses are removed rather than truncated or split.Source: §3.4 · p. 9

4 Post-Training

After visual-language alignment and multimodal mid-training, LoopVL continues through supervised fine-tuning and visual reinforcement learning. Fine-tuning trains responses with a teacher-forced objective, and reinforcement learning uses answer-correctness rewards with a progressive generation-length schedule.

4 Post-Training

  • Post-training follows visual-language alignment and multimodal mid-training with supervised fine-tuning and visual reinforcement learning.Source: Figure 5 · p. 9, §4 · p. 9

4.1 Supervised Fine-Tuning

  • Supervised fine-tuning uses 16.81B tokens from four data sources and trains conditional responses with teacher-forced mean cross-entropy over valid response tokens.The image and instruction form a bidirectional conditioning prefix; response positions can attend to that prefix and preceding response tokens.Source: §4.1 · pp. 9–10

4.2 Visual Reinforcement Learning

  • Visual reinforcement learning applies GRPO with answer-correctness rewards and raises the generation limit from 2048 to 4096 tokens while reducing candidates per input.GRPO computes group-normalized advantages and updates the policy with probability-ratio clipping and KL regularization against a reference policy.Source: §4.2 · p. 10

5 Evaluation

LoopVL is evaluated across 16 benchmark families using mostly short-answer greedy generation; the section compares overall performance, recurrence schedules, and evolving visual computation.

5.1 Results

  • LoopVL-1B shows strong visual-reasoning and general-understanding performance while remaining competitive on hallucination, chart-understanding, and scientific-reasoning benchmarks.Table 2 compares LoopVL-1B with compact vision–language models across 16 benchmark families; the supplied passages do not report individual score values.Source: Table 2 · p. 11, §5.1 · p. 11

5.2 Ablations

  • H2L3 achieves the highest score on all five benchmarks, while H2L1 outperforms H1L3 despite both executing 64 Transformer-layer calls.The comparison indicates that performance depends on how computation is organized between low-level state refinement and high-level state updates, not only on executed depth.Source: §5.2 · p. 12, Table 3 · p. 12

5.3 Loop Depth

  • The training-matched H2L3 inference schedule scores highest on all five benchmarks; changing schedules or adding calls does not reliably preserve or improve this checkpoint’s performance.With two model cycles fixed, increasing L invocations improves all five tasks; MMStar rises from 29.40 under H2L1 to 63.47 under H2L3. More cycles can instead reduce scores: MMStar falls from 63.47 under H2L3 to 58.20 under H3L3 and 24.00 under H4L3.Source: §5.3 · p. 12

5.4 Visual Aha Moment

  • The model-cycle transition coincides with a marked change in visual-attention organization, termed a Visual Aha Moment, and the pattern persists under tested backpropagation-route and visual anchor–gate changes.Entropy and Gini trajectories align the shift with the cycle boundary; the reported robustness checks reproduce the cross-cycle shift and boundary concentration patterns.Source: §5.4 · pp. 13–14

5.5 Cross-Loop Token Relevance

  • Selected examples show visual-token ranks changing across cycles, including target-region tokens moving from the first-cycle bottom half into the second-cycle top quartile.Natural-image examples illustrate question-related spatial reallocation, and a recorded example shows the same shared H layer assigning more attention to a target region under a later recurrent state; these examples do not estimate prevalence or establish improved accuracy.Source: §5.5 · pp. 15–16

5.6 Visual Attention Allocation

  • At matched endpoints, second-cycle L invocations have lower mean spatial entropy, while the final H layer has lower entropy and coverage and higher Gini concentration.These are averages over the 32-sample diagnostic collection; the measures use different query scopes and do not establish a universal per-sample trend or task relevance of concentrated locations.Source: §5.6 · p. 16

5.7 State Refinement and Prediction Formation

  • Visual states continue to update through the second cycle, while the final H invocation brings intermediate output distributions rapidly closer to the final prediction distribution.Linear recoverability of initial visual features changes across module calls and is lower at the recurrent endpoint than immediately after the projector; the state-change measures do not directly show how updates contribute to answers.Source: §5.7 · pp. 17–19

6 Related Work

The discussion separates recurrence schedules, carried states, visual interfaces, and evaluation, distinguishing LoopVL’s inherited backbone from its multimodal interface and spatial diagnostics.

6.1 Looped Transformers

  • Prior loop designs vary sharing granularity and execution schedules; LoopVL’s distinct contribution is multimodal construction with aligned spatial observations, not repetition itself.The discussion distinguishes shared-depth systems and adaptive depth from LoopVL’s particular visual-language construction.Source: §6.1 · pp. 19–20

6.2 Hierarchical and Nested Recurrence

  • LoopVL inherits HRM-Text’s L/H identities and nested schedule (Wang et al., 2025, 2026); its inference sweep changes execution, unlike independently trained H/L ablations.The paper distinguishes this hierarchical foundation from its visual interface and evidence-allocation diagnostics.Source: §6.2 · p. 20

6.3 Visual–Language Interfaces and Spatial Structure

  • Unlike Flamingo’s gated cross-attention and BLIP-2’s Querying Transformer (Alayrac et al., 2022; Li et al., 2023), LoopVL maps patch features into language space for recurrent updates.Spatial encoding, interface calibration, visibility, and training supervision are separate design choices, not one generic vision-strength explanation.Source: §6.3 · p. 21

6.4 Interpreting Recurrent Computation

  • Aligned visual slots enable same-layer, cross-invocation attention comparisons, but attention is an allocation diagnostic, not a causal explanation; “visual aha” names spatial evidence reallocation, not psychology.The paper also distinguishes representation changes, intermediate answers, and causal accounts of successful reasoning.Source: §6.4 · pp. 21–22

6.5 Training Regimes and Compute Accounting

  • Equal parameter storage does not imply equal FLOPs or runtime, LoopVL does not inherit Loopie’s cost results, and cumulative-data claims require stage manifests.The paper separates training regimes and data accounting from parameter-sharing descriptions.Source: §6.5 · pp. 22–23

6.6 Iterative Reasoning and Generalization

  • Prior prefix-sum, maze, and chess work reports harder-instance gains from added recurrences, alongside overthinking risks (Schwarzschild et al., 2021); LoopVL’s sweep establishes no such extrapolation.The sweep records a fixed H2L3-trained model under alternative execution schedules; it does not establish adaptive stopping or unlimited test-time scaling.Source: §6.6 · p. 23

7 Future Work

The study’s future-work agenda is bounded by limited mathematical-reasoning post-training, the absence of a matched Model-Loop control, reliance on one hierarchical framework, and image tasks with a frozen visual encoder.

7 Future Work

  • Post-training focuses primarily on mathematical reasoning, and limited data can lead extended reasoning to repeat tokens or segments and fail to complete a valid solution.Future work proposes broader tasks, more and more diverse post-training data, and richer preference and reasoning supervision.Source: §7 · p. 23
  • Without a matched control using a more general Model-Loop architecture, the study cannot establish whether Visual Aha Moments are intrinsic to Model-Loop computation.The authors also plan to test larger scales and more general architectures.Source: §7 · p. 23
  • Because LoopVL uses the HRM-Text hierarchical framework, the results characterize recurrent visual computation in that framework rather than the broader recurrent-architecture design space.Planned directions include alternative organizations, variable-depth computation, adaptive schedules, and inference-compute-matched settings.Source: §7 · p. 24
  • The study primarily covers image-based tasks with a frozen visual encoder, leaving multi-image, video, interactive, and agentic settings unexplored.Future work also proposes joint adaptation of the visual encoder and recurrent backbone.Source: §7 · p. 24

8 Conclusion

The conclusion presents LoopVL as a recurrent vision-language model built with nested loops and a visual interface, and frames visual aha moments as an interpretable account of recurrent evidence use while reserving stronger claims for controlled evaluation.

8 Conclusion

  • LoopVL combines nested Module-Loop and Model-Loop computation with dynamic grids, calibrated embeddings, and compatible spatial positions.Its language backbone is pretrained using the HRM-Text open-source framework and data.Source: §8 · p. 24
  • The analysis separates schedule sensitivity, attention concentration, cross-loop relevance reallocation, state updates, and final-readout alignment.These are presented as distinct aspects of the model’s recurrent visual computation.Source: §8 · p. 24
  • Visual aha moments offer a spatially interpretable description of recurrent evidence use, while stronger causal and generalization claims await controlled evaluation.The paper frames LoopVL as a way to examine changes when shared modules process visual evidence again.Source: §8 · p. 24

A Evaluation Settings and Scoring Methods

The appendix specifies the evaluation pipeline, model configuration, and scoring conventions used to assess LoopVL across benchmarks.

A Evaluation Settings and Scoring Methods

  • Evaluation uses a unified inference and scoring pipeline, with VLMEvalKit handling dataset loading and prompt construction for some benchmarks.Most evaluations use direct-answer prompts, greedy decoding, batch size 1, and a 32-token generation budget, with selected fixed-budget exceptions.Source: A Evaluation Settings and Scoring… · p. 29
  • Scoring uses task-specific rules, including relaxed accuracy for ChartQA and category-specific plus pooled F1 for POPE.HallusionBench averages aAcc, fAcc, and qAcc; MMEval-Pro requires all three questions in a triplet correct and macro-averages genuine accuracy across three source datasets; MMMU-Pro uses 1,730 questions.Source: A Evaluation Settings and Scoring… · p. 29
  • The main H2L3 model executes 128 layers per forward pass, and the H/L sweep tests 12 recurrence schedules using the same checkpoint.The execution order is LLLHLLLH; the final H-module output feeds the shared output head, and decoding disables the KV cache.Source: A Evaluation Settings and Scoring… · p. 29

B Diagnostic Metrics and Sample Sources

The appendix defines the samples and calculations used for attention, case-study, and state/output diagnostics.

B Diagnostic Metrics and Sample Sources

  • Attention analyses use 32 examples across four visual question-answering families, averaging spatial entropy and coverage over queries, heads, and examples.Coverage is the fraction of visual tokens needed to accumulate 80% of attention mass; Gini measures concentration after aggregating instruction and reference-answer attention within each head.Source: B Diagnostic Metrics and Sample Sources · p. 29
  • Case-study heatmaps average reference-answer attention over the visual grid; promoted tokens move from at or below the first-cycle median to at or above the second-cycle 75th percentile.Promotion rate is the share of second-cycle high-attention tokens that are promoted.Source: B Diagnostic Metrics and Sample Sources · p. 29
  • State and output probes use 16 examples each from VMCBench and AI2D, tracking visual-state changes and first-answer-token predictions across layer executions.The logit lens uses the corresponding stack’s final normalization and shared output head; KL divergence uses the final prediction distribution as reference.Source: B Diagnostic Metrics and Sample Sources · p. 29

C Additional Visual Reallocation Examples

Figures 12–16 show 40 additional selected RealWorldQA examples illustrating spatial reallocation in token-grid form. These qualitative cases do not estimate how prevalent reallocation is or establish that it improves answer accuracy.

C Additional Visual Reallocation Examples

Loading 2609.38426v1…