PaperlayerView exact revision on arXiv

Source-linked AI summary

Loop to Look Better: Learning Adaptive Depth for Question-Guided Perception in Vision-Language Models

Wencheng Ye, Anning Hu, Hongye Fang, Tianyi Wang, Junchi Yan

arXiv:2610.30824v1cs.CV

TL;DR

VLMs typically acquire visual tokens in a single, task-agnostic encoder pass, even as they gain more reasoning computation. LoopLook adds question-guided recurrent visual refinement with adaptive depth. It consistently improves performance across backbones and benchmarks, with gains up to 8.6% on MMStar and 8.1% on HallusionBench over the base model.

  • Problem

    VLM visual evidence is typically acquired in one task-agnostic pass, so later reasoning may lack task-critical details that were inadequately captured.

    Source: §1 · p. 1

  • Method

    LoopLook reuses a middle-to-late vision-encoder window for question-conditioned recurrent updates, with learned stopping and token-wise Mixture-of-Looks over loop states.

    Source: §1 · p. 2

  • Results

    LoopLook consistently improves average performance across four backbones and eight benchmarks, with gains of up to 8.6% on MMStar and 8.1% on HallusionBench over the base model.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LoopLook progressively refines attention toward queried evidence and allocates greater recurrent depth to fine-grained and hallucination-prone tasks.

    Source: §1 · p. 2

Abstract

from arXiv · show

Visual perception should be iterative and task-directed, with representations refined in response to the question and the need for further processing. While Vision-Language Models (VLMs) increasingly support extended reasoning, their visual tokens are still typically acquired in a single, task-agnostic encoder pass. This creates an evidence-acquisition bottleneck: models may reason longer without refining the visual evidence needed to answer the question. We propose LoopLook, a lightweight encoder-side recurrent perception framework that equips pretrained VLMs with adaptive visual depth. LoopLook recurrently revisits a middle-to-late window of the vision encoder, injecting question representations at each loop to guide visual refinement. A learned stopping mechanism adaptively determines the recurrent depth, while a token-wise Mixture-of-Looks (MoL) combines representations from different loops, allowing each visual token to selectively integrate information acquired across recurrent steps. Across four backbones and eight benchmarks, LoopLook consistently improves performance over the baseline, yielding gains of up to 8.6% on MMStar and 8.1% on HallusionBench. Our analysis shows that LoopLook progressively refines visual attention and adaptively allocates depth according to task difficulty. It also better preserves task-relevant visual information under reduced input resolution, while increasing time to first token (TTFT) by only 5.1-6.5%. By bringing recurrence from reasoning back into perception, LoopLook transforms visual perception from a one-shot glimpse into an adaptive process of recurrent refinement.

1. Introduction

VLMs can spend more computation reasoning while their visual evidence remains fixed after a single, task-agnostic encoder pass. LoopLook adds question-guided recurrent refinement with adaptive depth, improving performance across backbones and benchmarks.

1. Introduction

  • Visual perception can be refined iteratively according to the task, as successive observation can clarify ambiguous regions and relevant cues.The paper frames its question as how VLM visual processing can be guided by the question and extended adaptively when needed.Source: §1 · p. 1
  • The evidence-acquisition bottleneck is that visual tokens are fixed after one task-agnostic pass, even when later reasoning needs details the encoder did not capture.This is especially consequential for fine-grained perception and image-dependent reasoning, where additional reasoning cannot compensate for underprocessed visual information.Source: §1 · pp. 1–2
  • LoopLook reuses a middle-to-late encoder window, conditioning each recurrent update on the instruction, current visual summary, and loop step.A learned exit gate chooses whether to continue, and token-wise Mixture-of-Looks combines accumulated loop states; LoRA and trainable modulation enable lightweight adaptation with the language model frozen.Source: §1 · p. 2
  • LoopLook consistently improves average performance across four backbones and eight benchmarks, with gains of up to 8.6% on MMStar and 8.1% on HallusionBench over the base model.The framework also increases time to first token by only 5.1–6.5%.Source: §1 · p. 2
  • LoopLook progressively concentrates visual attention on queried evidence, allocates more depth to fine-grained or hallucination-prone tasks, and better preserves relevant information at reduced resolution.Its additional computation raises time to first token by 5.1–6.5% while leaving autoregressive decoding essentially unchanged.Source: §1 · p. 2

2. Related Work

Related work addresses visual evidence extraction through language-conditioned features and improved feature selection or delivery. LoopLook connects these directions with recurrent, question-conditioned visual encoding and lightweight adaptation of pretrained models.

2. Related Work

  • Prior work conditions visual feature extraction on language or improves how visual features are selected and passed to the language model.Examples include question-aware encoders, token packing, multi-layer aggregation, layer fusion, and routing.Source: §2 · p. 2
  • Recurrent computation has used shared-weight updates and learned halting, with explorations extending looped architectures to vision and multimodal modeling.The related-work discussion names block-recurrent designs, reViT, and LoopVL as examples.Source: §2 · pp. 2–3
  • LoopLook connects question-conditioned feature extraction with recurrence, while requiring lightweight adaptation rather than training the architecture from scratch.The paper says this preserves pretrained visual representations while enabling iterative refinement.Source: §2 · p. 3

3. Method

LoopLook adds recurrent, question-conditioned processing within a pretrained vision encoder, combining adaptive stopping with token-wise fusion across recurrent states. Its training objective jointly optimizes answer generation and recurrent depth.

3. Method

  • LoopLook combines a shared recurrent encoder window, loop-dependent feature modulation, Mixture-of-Looks, and adaptive stopping to refine visual representations before answer generation.The recurrent window is reused across loops, with its modulation state recomputed as the visual state evolves.Source: §3 · p. 3

3.1. Preliminary

  • Standard VLMs encode an image into visual tokens that remain fixed during language generation, before the model can reason about task-specific evidence.LoopLook addresses this with further encoder-side processing using recurrent perception with shared parameters.Source: §3.1 · pp. 3–4

3.2. Recurrent Visual Encoding

  • LoopLook reuses a middle-to-late encoder window across loops, placing recurrence within a coherent phase of aligned visual transformations.The readout tail and projector map each recurrent state back to the standard VLM interface.Source: Figure 3 · p. 3, §3.2 · p. 4

3.3. Task-Conditioned Modulation

  • At each loop, instruction context, the pooled current visual state, and a loop-step embedding condition affine modulation of normalized visual features.The shared window is reused with a recomputed modulation state as visual evidence evolves.Source: §3.3 · p. 4

3.4. Mixture-of-Looks

  • Mixture-of-Looks uses an exit gate to determine execution depth and a token-wise router to combine recurrent states for each visual token.The router fuses all recurrent states during training and only the executed prefix at inference; its stopping prior provides a sample-level preference over loops.Source: §3.4 · p. 4

3.5. Training Objective

  • Training jointly optimizes next-token prediction and expected recurrent depth, using a gradual penalty ramp that first emphasizes useful visual updates.The next-token loss trains recurrent states, routing, and the exit gate; the depth term encourages a lower expected loop count under the stopping prior.Source: §3.5 · pp. 4–5

4. Experiments

Across four VLM backbones and eight benchmarks, LoopLook improves performance over baselines and uses recurrent depth selectively. Experiments also examine visual refinement, reduced-resolution robustness, runtime, component contributions, and recurrent-window placement.

4.1. Experimental Setup

  • Experiments use four pretrained VLM backbones, eight benchmarks, and a middle-to-late recurrent encoder window; training updates 0.16%–0.40% of total parameters per backbone.The evaluation covers general understanding, perception and reasoning, and hallucination robustness; inference uses a maximum depth of four loops and an exit threshold of 0.4.Source: §4.1 · p. 5

4.3.1. Adaptive Looping Behaviors

  • LoopLook progressively concentrates visual attention on queried evidence, while its exit policy assigns more loops to fine-grained and hallucination-prone benchmarks than to easier recognition tasks.Token-wise routing favors early states for background tokens and later states near question-relevant regions, so localized evidence can use deeper representations without routing the whole image to the last loop.Source: §4.3.1 · pp. 5–7, Figure 4 · p. 7, Figure 5 · p. 7

4.2. Main Results

  • LoopLook improves average performance across four backbones by up to 4.6%; on Qwen2.5-VL-3B it gains 8.6% on MMStar and 8.1% on HallusionBench, while LLaVA-1.5-7B scores 48.5 average against existing enhancement methods.On LLaVA-1.5-7B, this improves the baseline by 3.3% and gives LoopLook the strongest performance on five benchmarks.Source: §4.2 · p. 5, Table 1 · p. 6, Table 2 · p. 6

4.3.1. Adaptive Looping Behaviors

  • At 50K resolution, Qwen2.5-VL-3B LoopLook reaches 54.9 MMStar overall accuracy versus 48.6 for Base, with a 12.4% fine-grained margin; average loops fall from 2.3 at 50K to 1.6 at 450K.LoopLook retains higher accuracy than the single-pass base at all tested resolutions and allocates more computation when visual evidence is weaker.Source: §4.3.1 · pp. 5–6, Table 3 · p. 6

4.3.2. Computational Efficiency

  • Encoder-side recurrence increases TTFT by only 5.1%–6.5%; decoding latency remains essentially unchanged and end-to-end latency is nearly identical because decoding dominates generation time.The exit-threshold curve saturates after τ = 0.4, so larger thresholds add computation for only marginal MMStar accuracy gains.Source: §4.3.2 · pp. 7–8, Table 4 · p. 8, Figure 7 · p. 8

4.4.2. Recurrent Window Placement

  • The middle-to-late CKA recurrent window performs best, exceeding Early-Mid, Early, and Cross-block windows by 0.7, 1.3, and 5.3 points, respectively.The cross-block window averages 63.2; the authors suggest a coherent shared transformation better aligns recurrent updates with pretrained layer functions.Source: Table 6 · p. 8, §4.4.2 · p. 8

4.4.1. Contribution of Individual Components

  • LoopLook reaches 68.5 average accuracy, above Param-Matched FT at 61.5 and FLOPs-Matched FT at 63.7; removing recurrence, task conditioning, or token-wise fusion lowers performance.The respective score drops are 4.4 points, 4.7 points, and 2.3 points; fixed-depth execution reaches 67.9 at two loops and 68.0 at three, versus 68.5 adaptively at 1.89 average loops.Source: §4.4.1 · p. 8

5. Conclusion

LoopLook adds task-conditioned recurrent visual encoding to pretrained VLMs, combining a shared encoder window, adaptive stopping, and token-wise feature integration. The appendix analyzes encoder blocks and window placement, reports qualitative attention and routing cases, and details the implementation.

5. Conclusion

  • LoopLook combines a shared encoder window, question-guided modulation, adaptive stopping, and token-wise Mixture-of-Looks to refine visual representations with lightweight additional training.The conclusion reports consistent performance improvements, adaptive depth allocation, reduced-resolution robustness, and modest latency overhead.Source: §5 · p. 9

A.1. Layer Similarity and Visual Blocks; A.2. CKA Maps Across Backbones

  • Across visual towers, CKA maps show coherent diagonal blocks; the analysis uses these phases to guide recurrent-window placement, and the selected blocks vary by architecture.Figure 8 presents the maps, while Table 7 identifies layer ranges selected for recurrent processing.Source: §A.2 · p. 13, §A.1 · p. 13, Table 7 · p. 13

A.3. Window Sweep Protocol

  • Windows inside coherent middle-to-late blocks give more stable gains, whereas very early windows and windows crossing strong block boundaries are less reliable.Candidate windows were compared under matched training and inference settings.Source: §A.3 · p. 13, §A.1 · p. 13

B. More Case Studies; B.1. BabyVision Open-Ended Cases

  • BabyVision case studies include open-ended counting and subtraction questions answered correctly by LoopLook.The subtraction example counts two planet groups separately before forming the equation.Source: B. More Case Studies · p. 13, §B.1 · p. 13

B.2. Attention Refinement Cases

  • In representative failure-to-success cases, LoopLook attention progressively concentrates on evidence regions, using two loops for easier localization cases and three for harder ones.Figure 11 contrasts base-model and recurrent-step attention distributions.Source: §B.2 · p. 13

B.3. Per-Token Loop Tier Visualization

  • The per-token routing visualization assigns each image token to the loop state with its largest routing weight.Figure 12 visualizes the token-wise Mixture-of-Looks decision.Source: §B.3 · p. 13

C.1. Backbones and Trainable Parameters

  • Across four backbones, training adapts fewer than 0.4% of total parameters; the language model remains frozen while encoder-side LoopLook modules are fully trainable.LoRA adapts the vision encoder, with trainable budgets ranging from 10.1M to 14.9M parameters.Source: §C.1 · pp. 13–14

C.2. LoopLook Hyperparameters

  • LoopLook uses a maximum depth of four loops and an inference exit threshold of 0.4, with mean-pooled visual summaries feeding modulation and exit components.The token-wise router and exit head are two-layer MLPs with hidden width d/4; the exit head outputs a sigmoid score after each loop.Source: §C.2 · p. 14

C.3. Training Setup

LoopLook is trained on Visual CoT data for one epoch using AdamW.

C.3. Training Setup

  • LoopLook trains for one epoch on 100K Visual CoT samples with global batch size 8, using AdamW and gradient clipping at norm 1.0.The learning rate is 3 × 10^-5, weight decay is 10^-2, and training runs on eight RTX 5090 GPUs.Source: §C.3 · p. 14

C.4. Evaluation Settings

The evaluation setup uses matched benchmark prompts and answer extraction protocols, with decoding settings specified for the baselines and LoopLook.

C.4. Evaluation Settings

  • Base and CoT baselines use greedy decoding with max_new_tokens=512; Self-Consistency samples five CoT responses at temperature 0.5 and selects by majority vote.LoopLook uses the corresponding backbone evaluation’s benchmark prompts and answer extraction protocol.Source: §C.4 · pp. 14–15
  • Figure 9 depicts a correct open-ended counting case in which LoopLook decomposes stacked cubes into layers and reaches a minimum count of 13.Source: Figure 9 · p. 15
  • Table 8 summarizes the trainable-parameter composition of LoopLook in millions.Source: Table 8 · p. 15

C.5. Efficiency Measurement

Efficiency measurements compare Base and LoopLook under matched GPU, example, and decoding conditions. The protocol reports TTFT, decoding latency, and end-to-end latency.

C.5. Efficiency Measurement

  • Efficiency tests use 200 randomly sampled benchmark examples on one RTX 5090 GPU, with five warm-ups and interleaved model timing to mitigate measurement bias.CUDA operations are synchronized at timing boundaries.Source: §C.5 · p. 15
  • For each resolution, the evaluation generates 64 tokens per example and reports TTFT, average per-token decoding latency, and end-to-end generation latency.The measurements are repeated and reported as mean latency across runs.Source: §C.5 · pp. 15–16
  • Timing runs use Python 3.13.5, PyTorch 2.14.0+cu130, CUDA 13.0, cuDNN 9.24, Transformers 5.16.1, and NVIDIA driver 580.95.05.Source: §C.5 · p. 16

D.1. Shared Training and Evaluation Settings

The section specifies controls for isolating recurrent processing, task conditioning, token-wise fusion, and adaptive stopping, alongside parameter- and FLOPs-matched fine-tuning. It also defines shared evaluation settings for the controls.

D.1. Shared Training and Evaluation Settings

  • Controls on Qwen2.5-VL-3B target recurrent processing, task conditioning, token-wise readout, adaptive stopping, and additional fine-tuning, with MMStar, V∗, HallusionBench, and their arithmetic mean reported.Training-based controls start from the same pretrained initialization; fixed-depth variants reuse the full LoopLook checkpoint and change only inference policy.Source: §D.1 · p. 16

D.2. Param- and FLOPs-Matched Fine-Tuning

  • The parameter-matched baseline retains a one-pass visual path and uses visual LoRA, with approximately 15.3M trainable parameters versus 14.9M for LoopLook.It is trained for 1.5× as many optimization steps, giving it a slightly larger training compute budget than LoopLook’s estimated overhead requires.Source: §D.2 · pp. 16, 19

D.3. w/o Recurrence

  • Setting Rmax = 1 removes recurrent execution during training while retaining the task representation, initial visual state, modulation networks, and first-step embedding.The control compares a single task-conditioned visual update with recurrent refinement.Source: §D.3 · p. 19

D.4. w/o Task Conditioning

  • The task-conditioning control sets the visual-side task vector to zero while retaining the visual-state conditioning, loop-step embeddings, adaptive exit, token-wise MoL, and training objective.The language model still receives the original instruction and supervision.Source: §D.4 · p. 19

D.5. w/o Token-Wise Fusion

  • The w/o token-wise fusion control replaces token-specific weights with shared weights, applying the same loop distribution to every token while leaving the scoring network and parameter count unchanged.The shared distribution can still vary across images.Source: §D.5 · p. 19

D.6. Fixed-Depth Execution (w/o Adaptive Exit)

  • Fixed-depth inference runs every sample for a chosen K from 1 to 4 loops, whereas the adaptive variant uses the exit gate to select K = ˆR.Both use the full checkpoint trained with Rmax = 4 and the same readout.Source: §D.6 · p. 19

E. LoopLook Training and Inference Algorithm

LoopLook unrolls recurrent steps during training and uses the exit gate to choose an inference prefix. Its router combines available loop states into visual representations for the projector and language model.

E. LoopLook Training and Inference Algorithm

  • Training unrolls all recurrent steps, while inference computes the prefix selected by the exit gate and routes available states through token-specific scores and the stopping prior.The fused representation passes through the tail and projector; training optimizes visual-side parameters with answer loss and a depth penalty.Source: E. LoopLook Training and Inference… · p. 19

F.1. Baseline Methods

The comparison includes three visual-side enhancement methods whose ways of integrating visual information differ. Their pathways also differ in backbone, token count, and input-resolution handling.

F.1. Baseline Methods

  • LoopLook is compared with Dense Connector, TGIF, and DeepStack as visual-side enhancement methods, with distinct pathways for multi-layer integration, instruction-guided fusion, and intermediate feature injection.The comparison is enabled by public checkpoints built on the Vicuna-7B-v1.5 language backbone, though Dense Connector and DeepStack use different visual recipes.Source: §F.1 · pp. 19, 21
  • Dense Connector concatenates features from three encoder-depth groups before the projector, using SigLIP-SO400M at 3842 resolution and producing 729 visual tokens per image.Its fusion is instruction-independent and its visual token count is fixed.Source: §F.1 · p. 21
  • TGIF uses the instruction to weight vision-encoder layers, combining the weighted features with penultimate-layer features before the standard projector.It retains CLIP ViT-L/14-336 and produces 576 visual tokens without changing encoder depth or token count at inference.Source: §F.1 · p. 21
  • DeepStack uses AnyRes-style multi-crop encoding and injects additional visual features into early language-model layers through residual connections.Its sequence includes a global view and local crops, and its token count varies with input resolution.Source: §F.1 · p. 21

F.2. Benchmark Details

The evaluation covers general visual understanding, fine-grained perception and reasoning, and hallucination robustness. It applies shared scoring rules and released benchmark prompts across settings.

F.2. Benchmark Details

  • All eight benchmarks use consistent prompts and answer extraction; multiple-choice items use option accuracy, open-ended items use reference-answer evaluation, and POPE and HallusionBench use yes/no answers.The benchmark prompts follow VLMEvalKit templates verbatim across evaluated settings.Source: §F.2 · p. 21
  • General-understanding benchmarks include MMStar, VMCBench, and AI2D, covering diverse visual tasks, heterogeneous VQA sources, and structured science diagrams.MMStar filters questions answerable from text alone; VMCBench uses its 1,000-sample DEV split.Source: §F.2 · p. 21
  • Perception and reasoning benchmarks cover fine-grained attributes, positions, and image text, basic visual abilities, and non-verbal puzzles involving patterns, spatial composition, and counting.These are V∗, BabyVision, and VisualPuzzles, respectively.Source: §F.2 · p. 21
  • POPE tests object hallucination across negative-example difficulty levels, while HallusionBench tests susceptibility to visual illusions, misleading associations, and conflicts between image evidence and language priors.Both use binary yes/no questions.Source: §F.2 · p. 21

F.3. Benchmark Prompts

  • All eight benchmarks use the prompt templates released in VLMEvalKit verbatim across evaluated settings.The image is prepended to the text prompt, and POPE and HallusionBench use their released annotation questions without an additional suffix.Source: §F.3 · pp. 21–22

Related papers

Loading 2610.30824v1…