PaperlayerView exact revision on arXiv

Source-linked AI summary

MoLE: Mixture of Latent Experts for Complementary Visual Reasoning

Yingcheng Liu, Tianyi Jiang, Yujuan Ding, jiangbo Ai, Xun Jiang, Guoqing Wang, Wei Ye, Yi Bin

arXiv:2610.01917v1cs.CVcs.AIcs.CL

TL;DR

Latent visual reasoning uses continuous intermediate states, but shared evidence and value projections can leave multiple latents redundant. MOLE specializes evidence access and transformations, then aggregates latent experts; across five benchmarks, it outperforms data-matched SFT and the strongest evaluated latent baseline at the same budget.

  • Problem

    Multiple latent tokens can access the same visual evidence through shared value projections, allowing redundant representations and leaving complementary extraction unspecified.

    Source: §1 · p. 1

  • Method

    MOLE routes evidence to isolated latent visual experts, applies expert-specific transformations, and aggregates their representations with summary experts.

    Source: §1 · pp. 2–3

  • Results

    78.6 average across five visual reasoning benchmarks, exceeding data-matched SFT by 4.9 and the strongest evaluated latent visual reasoning baseline at the same latent budget by 3.6.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    MOLE’s latents show lower similarity and more diverse attention, and masking the latent pathway reduces average performance by 9.2.

    Source: §1 · p. 2, §5 · p. 10

Abstract

from arXiv · show

Latent visual reasoning equips vision--language models with continuous intermediate states that can process visual evidence without explicit textual reasoning traces or repeated image operations. However, existing methods often allow multiple latent tokens to access the same visual evidence through shared value projections, providing no mechanism for them to extract complementary visual information; simply increasing the latent budget can therefore yield redundant latent representations. We argue that effective latent reasoning should encourage different latent tokens to extract complementary visual information, and thereby act as specialized visual experts. Based on this insight, we propose MoLE, a Mixture of Latent Experts framework that controls both what visual evidence each latent visual expert observes and how it transforms that evidence. MoLE isolates latent visual experts during evidence extraction and uses dedicated latent summary experts to aggregate the complementary representations of latent visual experts. A two-stage training pipeline first forces visual evidence through this latent pathway and then restores direct visual access, requiring neither predefined expert roles nor intermediate visual targets. Across five visual reasoning benchmarks, MoLE achieves an average score of 78.6, outperforming data-matched supervised fine-tuning by 4.9 and the strongest evaluated latent visual reasoning baseline at the same latent budget by 3.6. Representation analyses show lower latent-state similarity and more diverse visual attention, while masking the latent pathway reduces average performance by 9.2. These results demonstrate that specializing latent computation is more effective than merely increasing the number of latent tokens.

1 INTRODUCTION

MOLE addresses the risk that latent tokens redundantly process shared visual evidence by specializing what each observes and how it transforms it. It combines isolated latent visual experts, summary experts, and two-stage training; results show gains across five visual reasoning benchmarks.

1 INTRODUCTION

  • Shared visual evidence and value projections can make multiple latent tokens redundant, so increasing their count may add states without new reasoning capacity.Latent visual reasoning uses continuous hidden states for intermediate computation without explicit textual traces or repeated image operations.Source: §1 · p. 1
  • Specializing both the visual evidence each latent receives and its transformation encourages complementary information within a fixed latent budget.Figure 1 contrasts specialization with simply increasing latent count under shared evidence and transformations.Source: §1 · p. 2, Figure 1 · p. 2
  • MOLE routes visual tokens to isolated latent visual experts, applies expert-specific transformations, and uses latent summary experts to aggregate their representations.Answer tokens attend to both expert sets.Source: §1 · p. 2
  • Two-stage training first forces visual evidence through the latent pathway, then restores direct visual access, without predefined expert roles or intermediate visual targets.Training supervises answer tokens and uses router and ESVA diversity objectives.Source: §1 · pp. 2–3
  • 78.6 average across five visual reasoning benchmarks, exceeding data-matched SFT by 4.9 and the strongest evaluated latent visual reasoning baseline at the same latent budget by 3.6.Representation analyses find lower latent-state similarity and more diverse attention, while masking the latent pathway reduces average performance by 9.2.Source: §1 · pp. 2–3
  • The contributions include evidence from ablations, representation analyses, and interventions that learned latents are complementary and functionally used for answer generation.Source: §1 · p. 3

2 RELATED WORK

Related work spans text-based reasoning, interleaved visual observations, and continuous latent computation. MOLE addresses a gap in latent methods: they generally do not explicitly encourage complementary evidence extraction.

2 RELATED WORK

3 METHOD

Latent visual reasoning inserts continuous latent states between visual and text tokens, but does not itself specify which evidence each latent sees or how it transforms it. MOLE addresses this with routed, specialized experts, aggregation, and a two-stage training pipeline.

3.1 PRELIMINARIES

  • Latent states are inserted between visual and answer tokens, gather context during the prefix pass, and support autoregressive answer generation.The setup uses visual features, prompt embeddings, latent tokens, and answer embeddings in the decoder sequence.Source: §3.1 · pp. 3–4

3.2 MIXTURE OF LATENT EXPERTS

  • Each layer routes visual tokens to top-scoring latent visual experts, masking unselected expert–token connections so experts receive dynamically allocated evidence.The router computes expert logits and a distribution over experts for each visual token.Source: §3.2 · pp. 4–5
  • Expert-Specific Value Adapters transform routed visual values with expert-specific two-layer MLPs and a learned residual scale.The adapters apply only to routed expert–token connections.Source: Figure 2 · p. 4, §3.2 · p. 5
  • Latent summary experts aggregate visual experts’ representations, while answer tokens can attend to both sets; visual experts are isolated during evidence extraction.This avoids premature mixing of their representations before aggregation.Source: §3.2 · p. 5
  • Stage 1 blocks ordinary prompt and answer tokens from direct visual attention so latent experts learn task-relevant evidence; Stage 2 restores direct access for joint use.Source: §3.2 · p. 5
  • Answer-token language-modeling loss is combined with router load-balancing and z-losses and an ESVA diversity loss; both stages use the joint objective.The router losses prevent experts from becoming inactive and stabilize router-logit gradients; ESVA diversity penalizes redundant representations.Source: §3.2 · p. 6

4 EXPERIMENTS

Experiments compare MOLE with visual-reasoning models across five benchmarks and ablate its architecture, latent allocation, and training pipeline. MOLE leads the reported comparisons, while analyses support complementary latent representations and the value of the latent pathway.

4.1 EVALUATION BENCHMARKS

  • Evaluation covers five benchmarks with distinct perceptual demands, reporting six metrics: accuracy for all but POPE, which uses F1.CVBench contributes separate 2D and 3D accuracy metrics.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • The comparison includes a same-backbone, same-training-set SFT control and uses an eight-latent MOLE configuration with shared evaluation scripts for open models.LIVR is trained at latent budgets of eight and 16; proprietary results use different evaluation configurations and are reference-only.Source: §4.2 · p. 7, §4.3 · p. 8
  • 78.6 average across five benchmarks is the highest score; MOLE beats data-matched SFT by 4.9 and same-budget LIVR by 3.6.Even the strongest evaluated 16-step result remains 3.1 below MOLE with eight latents.Source: §4.2 · p. 7, Table 1 · p. 8

4.3 QUANTITATIVE ANALYSIS

  • MOLE has the lowest latent-state similarity and highest attention-pattern diversity, while masking latents lowers average performance by 9.2 despite direct visual access.Similarity is measured between latent hidden states; attention diversity compares their visual-token attention patterns.Source: Figure 3 · p. 8, §4.3 · p. 9
  • Removing ESVAs causes the largest performance drop; removing routing also hurts, and restoring causal mixing among visual experts degrades performance.The ablations respectively remove expert-specific transformations, equalize visual-token access, or permit direct causal interaction among visual experts.Source: §4.3 · pp. 7–8, Figure 3 · p. 8

4.4 ABLATION STUDIES

  • At fixed budgets of eight and 16, balanced visual and summary expert allocations perform best, while setting summary experts to zero reduces average performance by 2.9 and 2.4, respectively.The best 16-latent configuration does not outperform its eight-latent counterpart.Source: Table 2 · p. 9, §4.4 · p. 9
  • The complete two-stage training pipeline outperforms either stage alone, and combining the MoE and diversity losses yields the best performance.Stage 1 trains the latent pathway under a visual attention bottleneck before Stage 2 restores direct visual access.Source: Table 2 · p. 9, §4.4 · p. 9

5 CONCLUSION

The conclusion presents MOLE as a framework for complementary latent computation, using routed visual evidence, expert-specific transformations, and summary experts. It reports gains across five benchmarks and records the paper’s AI-use and reproducibility statements.

5 CONCLUSION

  • MOLE organizes latent visual reasoning around complementary computation, routing evidence to isolated visual experts, transforming it with ESVAs, and aggregating representations with summary experts.The conclusion frames this as an alternative to organizing reasoning around latent count.Source: §5 · p. 9
  • Across five benchmarks, MOLE improves data-matched SFT by 4.9 points on average and outperforms evaluated latent reasoning methods.The conclusion also reports lower latent-state similarity, more diverse visual-attention patterns, and useful latents under inference-time intervention.Source: §5 · p. 10

AI USE STATEMENT

  • AI-assisted tools were used only for minor language editing and polishing, and the authors take responsibility for the paper’s content.The authors state that they reviewed the resulting text.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The paper states that its complete anonymous code base is provided in supplementary material for reproducibility.MOLE is trained from Qwen2.5-VL-7B using a 63K-example subset constructed from CoVT-Dataset.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 METHOD DETAILS

This section introduces further implementation details for the ESVA and two-stage attention bottleneck.

A.1 METHOD DETAILS

  • The section provides further implementation details for the ESVA and the two-stage attention bottleneck.Source: §A.1 · p. 13

A.1.1 EXPERT-SPECIFIC VALUE ADAPTER

ESVA adds expert-specific transformations to a shared value projection, using layer-specific normalization and an expert-specific two-layer MLP. For routed connections, its output modifies the projected value used in attention aggregation.

A.1.1 EXPERT-SPECIFIC VALUE ADAPTER

  • ESVA retains the shared value projection for every expert and applies a layer-specific RMSNorm to its output.The normalization uses a learned layer-specific parameter and an epsilon term for numerical stability.Source: §A.1.1 · p. 14
  • Each latent visual expert applies a bias-free two-layer MLP with hidden dimension db to the normalized projected value.The MLP uses expert-specific matrices that are shared across decoder layers; the default db is 32.Source: §A.1.1 · p. 14
  • For a router-selected expert–visual-token connection, the MLP output is combined with the original projected value using a learned layer- and expert-specific scale.The resulting value replaces the standard value on that routed connection during attention-weighted aggregation.Source: §A.1.1 · p. 14
  • ESVA supplements rather than replaces the backbone value projection; the key projection remains unchanged, and other attention interactions use standard values.Source: §A.1.1 · p. 14

A.1.2 TWO-STAGE ATTENTION BOTTLENECK

The two-stage pipeline first requires visual evidence to pass through the latent pathway, then restores direct visual access so answers can use both latent representations and visual tokens.

A.1.2 TWO-STAGE ATTENTION BOTTLENECK

  • Stage 1 blocks direct visual access for prompt and answer tokens, forcing visual evidence to reach answers through the latent pathway.The latent visual experts remain isolated, while summary experts aggregate only their representations in both stages.Source: §A.1.2 · p. 14
  • Stage 2 starts from the Stage 1 checkpoint and restores direct visual access, letting answers combine learned latent representations with fine-grained visual evidence.Source: §A.1.2 · p. 14
  • Both stages optimize the same parameters with the same objective, and all MoLE-specific pathways remain unchanged.Source: §A.1.2 · p. 14

A.2 DATASET CONSTRUCTION

The training set contains 63K examples sampled from CoVT-Dataset; the appendix describes its sources, task-balanced sampling, and online decontamination.

A.2 DATASET CONSTRUCTION

  • The final 63K-example training set is constructed from CoVT-Dataset using task-balanced sampling and online decontamination.Source: §A.2 · p. 14

A.2.1 SOURCE DATASET

CoVT-Dataset combines vision-centric data with counting and relative-depth questions, spanning a broad set of visual tasks; its source records are highly imbalanced.

A.2.1 SOURCE DATASET

  • CoVT-Dataset combines vision-centric subsets of LLaVA-OneVision with a refiltered TallyQA subset and relative-depth questions derived from ADE20K.Source: §A.2.1 · p. 14
  • The dataset covers counting, compositional and spatial reasoning, visual mathematics, OCR, knowledge-based VQA, and detailed image description.Source: §A.2.1 · p. 14
  • 889,489 source records comprise 1,693,665 question–answer pairs, with TallyQA contributing more than 40% of records and several other sources substantially smaller.The reported imbalance motivates construction of a task-balanced subset.Source: §A.2.1 · p. 14

A.2.2 TASK-BALANCED SAMPLING

This appendix details task-balanced data preparation, model and evaluation configurations, metric calculations, and robustness, ablation, and inference-cost analyses.

A.2.2 TASK-BALANCED SAMPLING

  • Task balancing assigns 6,300 examples to each of ten families and stratifies samples by question format or answer category; online decontamination replaces discarded, invalid, or unreadable records.Source: §A.2.2 · p. 15, §A.2.3 · p. 16

A.2.3 ONLINE SAMPLE DECONTAMINATION

  • Decontamination checks exact image hashes, retrieves candidates using perceptual hashes and DINOv2 features, then verifies whole-image or local geometric matches before discarding a record.Source: §A.2.3 · p. 16

A.3 TRAINING CONFIGURATIONS

  • The data-matched SFT baseline uses three epochs of answer-token cross-entropy, while LIVR uses 8- or 16-token variants trained for two Stage 1 epochs and one Stage 2 epoch.Both use the same dataset and shared backbone/LoRA configuration described for the comparisons.Source: §A.3 · p. 16, §A.3.3 · p. 17, §A.3.4 · p. 17

A.5 CVBENCH SCORE CALCULATION

  • The appendix defines CVBench 3D accuracy from Depth and Dist. scores, describes latent-state similarity and attention-pattern diversity metrics, and reports best average performance at Router Top-K 2 and ESVA hidden dimension 32.Larger attention-pattern diversity values indicate more distinct visual-attention patterns; larger routing overlap and adapter dimensions did not improve performance.Source: §A.5 · p. 18, §A.6 · p. 18, §A.6.2 · p. 19

A.7.1 PARAMETER-MATCHED SHARED ADAPTER

  • 5.8 points separate MoLE from a parameter-matched shared-adapter model, which improves the backbone average by only 0.3; MoLE is better on every reported metric.Source: §A.7.1 · p. 20, Table 6 · p. 20

A.7.2 STABILITY ACROSS RANDOM SEEDS

  • 77.9–78.6 is the range of average scores across three random-seed runs, with a mean of 78.2 and standard deviation of 0.4.Every run beats the backbone on all six metrics; even the lowest average exceeds it by 5.4.Source: §A.7.2 · p. 20, Table 7 · p. 20

A.7.3 BACKBONE AND SCALE EXTENSION

  • 2.7, 1.3, and 1.5 points are the mean average-score gains over Qwen3-VL backbones at 2B, 4B, and 8B, respectively.Each run improves all six metrics, and average-score standard deviations are at most 0.3 at each scale.Source: §A.7.3 · pp. 20, 22, Table 8 · p. 21

A.7.6 INFERENCE COST

  • 0.84% more FLOPs and 1.55% more peak memory (0.245 GiB) accompany a 22.97% inference-time increase, or 0.086 seconds per example; average latency stays below 0.5 seconds.The reported average benchmark score gain over the backbone is 6.1 points.Source: §A.7.6 · p. 22, Table 10 · p. 22
Loading 2610.01917v1…