PaperlayerView exact revision on arXiv

Source-linked AI summary

Mixture of Layers: Dynamic Layer Routing for Visual Reasoning

Jeonghwan Kim, Sofia Stoica, Jiwan Chung, Ansel Blume, Hyeonjeong Ha, Zhenhailong Wang, Xin Luna Dong, Heng Ji

arXiv:2610.09440v1cs.CVcs.AI

TL;DR

Many MLLMs use fixed late-layer vision features, leaving access to fine-grained visual cues largely query-agnostic. MOL routes intermediate vision representations according to the instruction, and the paper reports improvements across fine-grained visual understanding benchmarks while remaining competitive on coarse-grained VQA.

  • Problem

    Many MLLMs rely on final- or penultimate-layer representations or static feature integration, although fine-grained tasks depend on cues such as small objects, localized text, and subtle attributes.

    Source: §1 · pp. 1–2

  • Method

    MOL uses instruction-conditioned routing to select intermediate vision-encoder representations at image, patch, or hybrid levels, including a reserve-layer mechanism.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Across fine-grained visual understanding benchmarks, MOL consistently improves over backbone MLLMs and prior static feature-integration methods while remaining competitive on conventional coarse-grained VQA tasks.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Routing patterns vary by task and granularity: image-level routing favors early layers, patch-level routing often selects later layers, and hybrid routing combines local evidence with broader context.

    Source: §4.3 · pp. 8–9

    MOL requires pretrained vision-encoder hidden states and, for text-conditioned variants, an aligned text encoder; routing choices may need retuning for substantially different encoders or settings.

    Source: A Limitations and Broader Impacts · p. 16

Abstract

from arXiv · show

Pre-trained vision encoders contain layer-wise visual representations that differ in spatial granularity, semantic abstraction, and sensitivity to local details. However, most Multimodal Large Language Models (MLLMs) rely on only the final or penultimate vision encoder representations or fixed aggregation rules, making visual abstraction largely query-agnostic and limiting access to fine-grained cues such as small objects, spatial details, text, and subtle visual attributes. In this work, we propose Mixture of Layers (MoL), an instruction-conditioned layer routing approach at the visual patch level that dynamically aggregates query-relevant latent representations from intermediate vision encoder layers. Given a text query, MoL predicts routing probabilities over vision encoder layers and performs a top-k sparse aggregation over selected hidden states at either the image level, patch level, or through a hybrid routing mechanism. In doing so, MoL enables query-adaptive access to layer-specific visual features for fine-grained visual reasoning. Our experiments across 7 fine-grained visual reasoning tasks demonstrate substantial performance improvements, especially across fine-grained visual grounding and understanding tasks such as +18.9% improvement on V* in overall accuracy, +4.5% on HRBench4K, and +16.3% on CharXiv compared to the baseline MLLMs, without resorting to multi-resolution inputs, simple interleaving of multiple vision encoders, or increasing the number of patch tokens. We study vision encoders' receptive field scales across different layers and their sampling behaviors to provide an in-depth analysis of why layer-wise sampling is helpful, demonstrating that conditional visual representations are a key step towards better visual perception and reasoning in MLLMs. Our project page is available at https://wjdghks950.github.io/mol.github.io/.

1 Introduction

Many MLLMs rely on fixed late-layer vision features, limiting query-agnostic access to fine-grained cues. Mixture of Layers (MOL) routes intermediate visual representations according to the instruction, with reported gains on fine-grained tasks.

1 Introduction

  • Fine-grained reasoning can depend on small objects, localized text, subtle attributes, counting, and spatial relations that may not be preserved in a single final-layer representation.The introduction argues that fixed late-layer representations provide limited, query-agnostic access to granular visual information.Source: §1 · pp. 1–2
  • MOL reports gains of 18.9% on V* and 16.3% on CharXiv, and improves fine-grained reasoning across seven benchmarks without adding visual tokens or relying on multiple vision encoders.Its routing variants select instruction-relevant intermediate representations at image or patch level, or combine both.Source: §1 · p. 2
  • MOL predicts instruction-conditioned routing distributions over vision-encoder layers and selectively aggregates intermediate patch representations through image-level, patch-level, or hybrid routing.The hybrid variant combines image- and patch-level routing and uses disagreement to gate a reserve layer.Source: §1 · p. 2
  • MOL’s contributions include a routing formulation for both image-level MOLlayer and patch-level MOLpatch decisions, plus a hybrid mechanism with a disagreement-gated reserve layer.The hybrid is described as balancing local fine-grained evidence with global semantic context.Source: §1 · p. 2

2 Related Work

Prior MLLMs and visual abstractors generally use fixed late-layer features, while intermediate-layer methods often sample statically or increase visual tokens. MOL instead routes intermediate features by instruction and representation granularity, targeting fine-grained perception without increasing the token budget.

2 Related Work

  • Most MLLMs and text-conditioned querying modules still operate on fixed final- or penultimate-layer representations, while visual abstractors transform embeddings already extracted from those layers.MOL instead routes across intermediate vision-encoder layers according to the instruction.Source: §2 · pp. 2–3
  • Prior intermediate-feature methods use rule-based or visual-feature-only sampling, whereas MOL supports instruction-guided sampling at patch and layer levels.TGIF predicts query-conditioned layer weights but averages all vision layers; MOLlayer uses sparse top-k routing based on query-layer compatibility.Source: §2 · p. 3
  • Fine-grained benchmarks target localized text, small objects, and subtle attributes that are not stressed by many widely used coarse-grained visual reasoning benchmarks.The passage names V*, HRBench, NaturalBench, RealWorldQA, and CharXiv as examples addressing this gap.Source: §2 · p. 3
  • Unlike approaches that increase visual tokens with multi-resolution inputs or concatenate features from multiple encoders, MOL refines each patch using instruction-conditioned intermediate-layer features.The passage positions MOL as complementary to existing multi-patch and high-resolution approaches.Source: §2 · p. 3

3 MIXTURE OF LAYERS: Selective Visual Representations for Fine-Grained Visual Reasoning

Mixture of Layers (MOL) routes instruction-relevant visual representations from intermediate vision-encoder layers instead of relying only on fixed late-layer features. It offers image-level, patch-level, and hybrid routing, with reserve-layer fallback and an option to route and fuse features from multiple encoders.

3.1 Preliminary

  • MOL uses aligned vision and text encoders, with the pooled text representation serving as a query to score visual-layer features for routing.Each vision layer supplies hidden states for visual tokens; the text embedding sequence is pooled into a query representation.Source: §3.1 · p. 3
  • A reserve layer provides a fallback when routing is uncertain or patch evidence is noisy, while a learned gate combines it with selected non-reserve layers.The reserve is the backbone’s last or penultimate layer; training uses autoregressive loss and an auxiliary load-balancing loss to encourage balanced layer use.Source: §3.1 · p. 4

3.2 MIXTURE OF LAYERS

  • MOLlayer makes one query-guided layer selection for the image, whereas MOLpatch makes a separate layer selection for each visual patch.MOLlayer average-pools each layer’s visual tokens; patch-level routing lets image regions draw from different layer abstractions without changing the output patch-token count.Source: §3.2 · pp. 4–5
  • MOLhybrid fuses global and local routing, using their disagreement to set how much each patch relies on the reserve layer.The hybrid routing distribution interpolates global and patch-level routing; greater disagreement corresponds to greater reserve-layer reliance.Source: §3.2 · p. 5
  • MOL can route within each of multiple vision encoders, then upsample and concatenate their projected representations before a learned linear projection produces the final patch sequence.The described fusion resizes the DINOv2 stream to the CLIP token-grid size before concatenation.Source: §3.2 · pp. 5–6
  • The section presents Table 1 as accuracy across backbones and router types, Table 2 as comparison with feature-integration methods, and Table 3 as single- versus multi-encoder routing.The supplied table passages describe scope and metrics but do not provide table-cell results.Source: Table 1 · p. 6, Table 2 · p. 6, Table 3 · p. 6

4 Experiments

Experiments evaluate MOL across multiple MLLM backbones, vision encoders, and fine-grained benchmarks, then analyze layer selection and routing ablations. MOL improves fine-grained reasoning while largely preserving coarse-grained VQA performance.

4 Experiments

  • The experiments use LLaVA-1.5-13B, Llama-3-8B-V, and Phi-1.5-1.3B with CLIP, DINOv2 w/ Txt, or SigLIP encoders, evaluating six fine-grained benchmarks.The setup uses the backbones' provided pre-training and instruction-tuning datasets for comparison.Source: §4 · p. 7

4.1 Effect of MIXTURE OF LAYERS on Model Perception

  • MOL improves fine-grained reasoning across backbones and encoders, including +13.44 on LLaVA-v1.5-13B V* overall and +18.90 for MOLhybrid on V* with DINOv2 w/ Txt.The reported gains also include direct attribute recognition (+20.87), OCR (+10.00), and CharXiv (+5.33) for the primary LLaVA-v1.5-13B setting.Source: §4.1 · p. 7
  • MOLpatch outperforms Dense Connector by 3.52 points on overall V*, while MOLhybrid improves performance on challenging benchmarks including GPT4V-hard, MMStar, and HRBench.MOL uses a single vision encoder and matches or outperforms prior multi-encoder approaches in the reported comparison.Source: §4.1 · p. 7
  • MOL largely preserves coarse-grained VQA, with small gains on POPE, GQA, and MMMU and minimal change on TextVQA.Table 4 reports coarse-grained VQA performance.Source: Table 4 · p. 7, §4.1 · p. 7

4.2 MIXTURE OF LAYERS with Multiple Vision Encoders

  • The dual-encoder system scores 90.34 on V* versus 82.35 for Interleaved-MoF, but reported comparisons are mixed and do not isolate routing's contribution.The comparison is between complete systems rather than a controlled routing-only ablation.Source: §4.2 · p. 8

4.3 Analysis of Sampling Behavior

  • MOLlayer favors early layers for global detail, MOLpatch often selects later contextual layers per patch, and MOLhybrid balances these behaviors.Receptive field increases with depth: early layers capture local interactions, while later layers aggregate broader context; attention maps also align selected layers with query-relevant regions.Source: §4.3 · pp. 8–9, Figure 3 · p. 9, §4.4 · p. 10

4.4 Ablations

  • Text-conditioned MOLhybrid improves V* by 13.44 points, whereas image-only MOLimage routing drops 9.16 points on V*; Table 5 compares both against baseline.The reported results support using the instruction as a signal for selecting relevant visual abstractions.Source: Table 5 · p. 9, §4.4 · p. 10
  • Removing the reserve layer degrades performance by 3–5 points across benchmarks, while small k values such as 1 or 4 are sufficient and often optimal.The reserve layer serves as a stable anchor when routing over intermediate layers is uncertain or noisy.Source: Table 6 · p. 9, §4.4 · p. 10

5 Conclusion

MOL routes instruction-relevant intermediate vision representations for fine-grained reasoning, improving fine-grained benchmarks while remaining competitive on conventional coarse-grained VQA. Its routing strategies use different parts of the vision-layer hierarchy, and the appendix roadmap identifies where supporting details and evidence appear.

5 Conclusion

  • MOL consistently improves over backbone MLLMs and prior static feature-integration methods on fine-grained visual understanding benchmarks while remaining competitive on conventional coarse-grained VQA.The conclusion presents dynamic selection of visual abstraction as an alternative to scaling visual inputs or adding vision encoders.Source: §5 · p. 10
  • Layer-level routing favors localized early layers, patch-level routing leverages contextualized later layers, and hybrid routing combines both regimes.The conclusion attributes these patterns to the different regions of the vision-encoder hierarchy used by each strategy.Source: §5 · p. 10
  • MOL dynamically selects intermediate vision-encoder representations conditioned on the textual instruction for fine-grained visual reasoning.The conclusion describes layer-level, patch-level, and hybrid routing for single- and dual-vision-encoder schemes.Source: §5 · p. 10

Appendix

  • The appendix roadmap directs readers to scope and limitations, model and evaluation procedures, extended empirical evidence, router and receptive-field analyses, qualitative examples, and reproducibility notes.It identifies Sections A–G as covering these materials.Source: Appendix · p. 15

A. Limitations and Broader Impacts

The section covers the scope of claims, methodological limitations, expected benefits, risks, and responsible-use considerations.

A. Limitations and Broader Impacts

  • The section addresses claim scope, methodological limitations, expected benefits, risks, and responsible-use considerations.The supplied passage names these as its topics.Source: A. Limitations and Broader Impacts · p. 15

B. Training Details

The appendix materials cover training setup, evaluation details, additional benchmark results, router and receptive-field analyses, qualitative examples, and reproducibility notes.

B. Training Details

  • Training details are identified as a subject of the appendix materials.The supplied roadmap does not provide the training settings themselves.Source: B. Training Details · p. 15

C. Evaluation Details

  • Evaluation details cover benchmark scripts, output files, score aggregation, and missing-entry conventions.The appendix also includes a full benchmark matrix, a k sweep, and a reserve-layer ablation.Source: C. Evaluation Details · p. 15

E. Router and RF Analyses

  • Router and receptive-field analyses cover layer-selection behavior, receptive-field overlays, router-use diagnostics, and DINOv2 placeholders.The appendix also lists selected attention panels and a LLaVA + DINOv2 heatmap analysis.Source: E. Router and RF Analyses · p. 15
  • The appendix includes remaining caveats, source artifacts, and checkpoint-configuration conventions.These are listed under the reproducibility notes.Source: E. Router and RF Analyses · p. 15

A Limitations and Broader Impacts

MOL is targeted at fine-grained visual reasoning, with especially strong gains on tasks requiring localized evidence and smaller or near-neutral gains on conventional coarse-grained VQA. Its applicability, evaluation, and interpretability have stated boundaries, while the appendix documents training and reproducibility details and the section outlines benefits, risks, and responsible-use considerations.

A. Limitations and Broader Impacts

  • MOL targets fine-grained visual reasoning: gains are strongest on localized-evidence tasks, while conventional or coarse-grained VQA gains are smaller and sometimes near neutral.Examples of localized-evidence tasks include V*Bench, CharXiv, HRBench, OCR-style questions, direct attribute recognition, and relative-position reasoning.Source: A Limitations and Broader Impacts · p. 16
  • MOL assumes access to pretrained vision-encoder hidden states and, for text-conditioned variants, an aligned text encoder; it is less directly applicable to encoder-free models or models exposing only final embeddings.Router parameters, k, and reserve-layer choices may need retuning for substantially different encoders, tokenization, resolutions, or domains.Source: A Limitations and Broader Impacts · p. 16
  • Most large-scale training settings have single-run results because training 8B–13B multimodal models is computationally expensive.The evaluation does not exhaust modern MLLM architectures, and the authors call for repeated runs, stronger uncertainty estimates, and additional domains.Source: A Limitations and Broader Impacts · p. 16
  • Router probabilities, receptive-field overlays, and selected-layer attention maps diagnose routing behavior but are not complete causal explanations of model answers or guarantees of faithful reasoning.Attention can be diffuse when a selected layer contributes useful features, and plausible-looking attention can coincide with an incorrect answer.Source: A Limitations and Broader Impacts · p. 16
  • Potential benefits include assistive image understanding, chart interpretation, accessibility, and scientific document analysis, without necessarily increasing visual tokens or requiring multiple vision encoders.The authors describe this as a potentially more compute-conscious path than scaling resolution or passing longer visual sequences.Source: A Limitations and Broader Impacts · p. 16
  • Improved fine-grained perception may raise privacy and surveillance risks, while MOL does not remove inherited biases or provide safety alignment for high-stakes deployment.The authors caution against treating it as sufficient for medical, legal, hiring, biometric-identification, or safety-critical robotics settings.Source: A Limitations and Broader Impacts · p. 16

Appendix

  • The Vicuna-13B + CLIP setting follows the LLaVA-v1.5 two-stage recipe, while Llama-3-8B + SigLIP and Phi-1.5 + SigLIP use the Bunny two-stage recipe with LoRA during instruction tuning.The reserve layer is the penultimate vision-encoder layer used as an anchor representation; appendix tables use normalized method names.Source: Table 7 · p. 17, Table 8 · p. 17

A. Limitations and Broader Impacts

  • The authors frame MOL as a research representation-routing method and recommend privacy-aware data handling, domain-specific evaluation, bias and robustness audits, and human oversight in high-stakes settings.They report releasing training details, normalized router names, checkpoint-configuration conventions, and analysis scripts to support auditing.Source: A Limitations and Broader Impacts · p. 17

B Training Details

The appendix records the hardware, optimization, and objective used for training, alongside the naming conventions and data paths for the model families.

B Training Details

  • Training used four NVIDIA H100 NVL GPUs with ZeRO-2 for connector pretraining and ZeRO-3 for multimodal instruction tuning, plus bf16, tf32, gradient checkpointing, and a cosine schedule.The stated warmup ratio was 0.03, weight decay 0, and maximum sequence length 2048.Source: B Training Details · p. 17
  • The objective combines standard autoregressive language-modeling loss with auxiliary router load balancing; the default coefficient is 10^-2 and router temperature anneals from 2.0 to 1.0.Router statistics are saved during training for Appendix E analysis.Source: B Training Details · p. 17
  • The patch_layer and hybrid implementation variants are named MOLpatch and MOLhybrid, while attn-token variants are grouped as MOLlayer.For each setting, the better-performing layer-level variant is reported.Source: B Training Details · p. 17
  • Vicuna/LLaVA training follows the LLaVA-v1.5 data mixture, while Bunny uses its pretraining and instruction-tuning data for Llama-3 and Phi backbones.Reported global batch sizes account for four GPUs and gradient accumulation.Source: B Training Details · p. 17

C Evaluation Details

The appendix describes evaluation outputs and scoring conventions, and identifies the scope and availability of its expanded benchmark tables.

C Evaluation Details

  • Each evaluation run writes predictions, per-example details, and a summary file.The described output location begins with playground/data/eval/BENCHMARK/answers.Source: C Evaluation Details · p. 17
  • Accuracy is the default metric unless a benchmark specifies otherwise; scores are percentages for the listed benchmarks, and multi-split scores use the official aggregate.The listed benchmarks include V∗Bench, HRBench4K/8K, MMStar, NaturalBench, RealWorldQA, CharXiv, and conventional VQA benchmarks.Source: C Evaluation Details · p. 18
  • Tables 9 and 10 provide the full benchmark matrix for the primary LLaVA-v1.5 family, with unavailable entries left empty and entirely empty columns omitted for selected rows.Table 10, part 2, identifies MME-RW as MME-RealWorld.Source: C Evaluation Details · p. 18, Table 9 · p. 18, Table 10 · p. 18

D Additional Benchmark Results

The appendix reports broader benchmark patterns, routing ablations, receptive-field analyses, and qualitative attention examples for MOL.

D Additional Benchmark Results

  • The broader LLaVA-v1.5 results show the strongest gains on fine-grained benchmarks, while conventional VQA-style benchmarks usually remain close to baseline.The expanded results appear in Tables 9 and 10.Source: D Additional Benchmark Results · p. 18
  • A small selected set of layers is usually sufficient, and increasing k does not automatically improve results; the best k varies by task and routing family.The table isolates the effect of the number of selected layers.Source: D Additional Benchmark Results · p. 18
  • Removing the reserve layer hurts V∗Bench, HRBench, POPE, and MMMU, supporting its role as a stable anchor representation.The ablation directly tests the reserve layer.Source: D Additional Benchmark Results · p. 18, Table 12 · p. 19

E Additional Router and Receptive Field Analyses

  • For DINOv2, MOLpatch often favors early layers for perceptual categories, while MOLhybrid assigns more weight to mid-to-late layers for broader reasoning categories.Figure 4 plots category-wise layer routing probabilities against receptive-field background; the pattern supports hybrid routing's use of local evidence and deeper context as needed.Source: E Additional Router and Receptive… · p. 19, Figure 4 · p. 19

F Additional Qualitative Examples

  • DINOv2 attention maps localize OCR, counting, and attribute cues to queried regions, including in cluttered scenes and semantically broader examples.Figure 5 visualizes selected-layer self-attention; the examples include text, jersey numbers, logos, and an instrument held by an ape.Source: F Additional Qualitative Examples · p. 19, Figure 5 · p. 20

G Reproducibility Notes

The appendix records reproducibility artifacts and scope caveats, while checklist responses describe planned releases and evaluation boundaries.

G Reproducibility Notes

  • The appendix identifies benchmark and analysis artifacts, with checkpoint config files treated as the source of truth for router settings and learning-rate metadata.The authors note that most large-scale settings lack multiple random-seed runs because training is computationally expensive, prioritizing breadth instead.Source: G Reproducibility Notes · p. 20

NeurIPS Paper Checklist

  • The authors state that the evaluation does not exhaust modern MLLM architectures and that broader backbones, modalities, and settings remain directions for extension.The checklist says the abstract and introduction scope claims around fine-grained visual reasoning and results in Section 4.Source: NeurIPS Paper Checklist · p. 21
  • The paper reports no theoretical results or formal proofs, and says LLMs are not used as a non-standard research assistant or data-generation component in its core methodology.Source: NeurIPS Paper Checklist · pp. 21, 25
  • The authors report that code and reproduction instructions will be released, with documentation, setup instructions, training and evaluation scripts, and license information.They state that datasets are publicly available and asset licenses and terms will be documented.Source: NeurIPS Paper Checklist · pp. 22, 24
  • The authors report no statistical significance tests or error bars, citing the expense of experiments; compute resource details are provided in the Appendix.Source: NeurIPS Paper Checklist · p. 23
  • The authors report using publicly available datasets and models without human subjects, and identify no direct societal impact beyond general effects of improving multimodal capabilities.Source: NeurIPS Paper Checklist · pp. 23–25

Related papers

Loading 2610.09440v1…