PaperlayerView exact revision on arXiv

Source-linked AI summary

Late-Layer Fusion is Enough: Dual-Path Vision Token Routing for Multimodal Large Language Models under Visual Saturation

Siyuan Liu, Jinyang Wu

arXiv:2606.09131v1cs.AIcs.CLcs.CVcs.LG

TL;DR

MLLMs commonly process image and text tokens symmetrically even though vision representations saturate earlier than text representations. The paper introduces DPVR-LF, which routes vision tokens through a shallow trainable branch, processes text deeply, and performs one final fusion. With 3% trainable parameters, it matches or exceeds full fine-tuning across eight benchmarks while reducing forward computation.

  • Problem

    MLLMs apply the same deep computation to image and text tokens despite evidence that vision tokens saturate earlier while text tokens continue benefiting from depth.

  • Method

    DPVR-LF routes vision tokens at a saturation split into a one-layer trainable side branch, runs a thirteen-layer text-only forward, and fuses both streams in the final layer.

  • Results

    3% trainable parameters let DPVR-LF match or exceed full fine-tuning across eight benchmarks while saving 25–30% of forward FLOPs.

  • Takeaways & Limitations

    A single final fusion layer can preserve strong perceptual competence while reducing repeated deep-layer computation for image positions.

  • Takeaways & Limitations

    Performance is weaker on BLINK and MMBench-CN, and decode-time implementation currently falls back to no-cache full-context forward steps.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) commonly inherit the deep, symmetric Transformer backbone designed for unimodal text modeling, and apply the same computation uniformly to image and language tokens. This design overlooks a key modality asymmetry: image and text tokens differ substantially in information density, redundancy, and required reasoning depth. Through a layer-wise analysis of LLaVA-1.5, we observe that vision tokens tend to saturate in the middle layers. Specifically, text-to-image attention decreases from 0.68 at layer 0 to 0.07 by layer 4, and stabilizes near 0.04 after layer 18, whereas text tokens continue to benefit from deep semantic processing. These findings suggest a mismatch between architectural symmetry and depth-asynchronous modality evolution, resulting in redundant visual computation and possible drift in perceptual representations during deep task-specific adaptation. Motivated by this, we propose Dual-Path Vision Token Routing (DPVR), a modality-asymmetric routing framework for efficient MLLMs. Its core instantiation, DPVR-LF (Late-Layer Fusion), routes vision tokens at the saturation point into a one-layer trainable side branch, runs a thirteen-layer text-only forward that skips image positions in the deep stack, and re-fuses the visual and textual streams only at the final layer. With approximately 3% trainable parameters, DPVR-LF preserves competitive multimodal performance on standard benchmarks while reducing visual computation in the deep Transformer stack. The results challenge the conventional assumption that vision tokens must traverse all deep language-model layers, and indicate that a single late fusion layer can be sufficient for maintaining strong perceptual competence in LLaVA-style MLLMs.

1. Introduction

Mainstream MLLMs process image and text tokens through the same deep Transformer stack, despite evidence that vision tokens saturate earlier while text tokens continue benefiting from depth. DPVR-LF responds by routing stabilized vision tokens through a shallow trainable branch and fusing them with deeply processed text only at the final layer.

  • The symmetric-architecture default: Standard MLLMs concatenate projected visual features with text tokens and apply identical computation throughout a deep decoder-only Transformer stack.This design assumes image and text tokens require the same layer-wise processing despite their modality differences.
  • Empirical evidence: visual saturation: Vision-token adjacent-layer cosine similarity stays above 0.92 from layer 0 onward, indicating marginal deep residual updates.This provides a complementary hidden-state measure of visual saturation.
  • Empirical evidence: visual saturation: Text-to-image attention falls from 0.68 at L0 to 0.07 at L4 and stabilizes near 0.04 thereafter.The ten-fold collapse indicates rapid visual disengagement from later textual processing.
  • Empirical evidence: visual saturation: Vision tokens reach prediction space at L22, one layer before text tokens at L23.The logit-lens transition further suggests asynchronous modality evolution.
  • Dual-Path Vision Token Routing: DPVR-LF branches vision tokens into a one-layer trainable side path, performs a thirteen-layer text-only forward, and reunites both streams in one final fusion layer.The 7B and 13B models use saturation splits at s=18 and s=28, respectively.
  • Empirical validation: With 3% trainable parameters, DPVR-LF matches or exceeds full fine-tuning across eight benchmarks while saving 25–30% of forward FLOPs.The reported measured A800 latency reduction is −28.0%.

2. Related Work

Related work spans multimodal architectures, visual-token reduction, conditional computation, parameter-efficient tuning, and cross-layer analysis. DPVR differs by structurally routing image tokens away from deep layers while retaining the existing backbone and training setup.

  • Multimodal large language models: Multimodal systems evolved from contrastive image-text alignment toward generative models connecting vision encoders to pretrained language models through learnable bridges.The cited generations include CLIP, Flamingo, and related multimodal architectures.
  • Token reduction in MLLMs: Visual-token reduction methods prune, merge, or discard image tokens after shallow processing, often heuristically or only during inference.Examples include FastV, VTW, LLaVA-PruMerge, and TokenPacker.
  • Architectural compute reduction and conditional compute: DPVR uses deterministic modality-conditional routing: image tokens skip a fixed deep segment while text tokens retain the full stack.Its structural 13-layer text-only segment differs from learned or stochastic conditional routing.
  • Cross-modal architectures and decoder-only design: Earlier cross-modal systems explicitly separate perception and reasoning with modules such as gated cross-attention, Q-Former, or resamplers.Decoder-only LLaVA-style designs instead favor training simplicity and ecosystem fit.
  • Parameter-efficient fine-tuning: LoRA, QLoRA, and adapter modules reduce trainable parameters through low-rank updates, quantization, or bottleneck layers.DPVR occupies a different PEFT point by changing modality-specific computation rather than distributing low-rank corrections.
  • Cross-layer analysis of language and vision-language models: Logit lens and tuned lens trace intermediate prediction emergence, while prior probing studies characterize per-layer linguistic competence.The paper applies related cross-layer analysis to visual and textual dynamics inside MLLMs.

3. Methodology

The methodology analyzes layer-wise modality dynamics and implements DPVR-LF, which routes saturated vision tokens through a trainable side branch while reserving deep computation for text and a final fusion step.

  • Visual saturation analysis: 500 LLaVA-665k samples are analyzed using adjacent-layer cosine similarity, text-to-image attention mass, and logit-lens KL divergence.The analysis probes vanilla LLaVA-1.5-7B without fine-tuning and records hidden states and attention weights at every layer.
  • Visual saturation analysis: 0.04 is the post-L18 text-to-image attention asymptote, while vision-token similarity is already ≥0.92 from L0 onwards.The split at layer 18 occurs after attention has collapsed but before the logit-lens prediction transition at L22.
  • Split selection: s∈[18,24] is a robust 7B split plateau, with six-benchmark mean accuracy varying within 0.1 pp.The 13B saturation transition shifts to L33, motivating a separate split-layer sweep rather than a simple depth-proportional rule.
  • DPVR-LF architecture: DPVR-LF routes vision tokens at s=18 through one trainable transformer block and runs a 13-layer text-only deep forward before final fusion.The 7B deep stack skips image positions in layers 18–30, then reassembles them for full attention at layer 31.
  • DPVR-LF architecture: A final fusion layer is required because fully text-only deep processing produces zero gradient to the visual side branch under LLaVA’s labeling convention.The final image-text attention path restores gradient flow through the image representation.
  • Compute and routing trade-offs: DPVR-LF removes image computation from 13 of 14 deep layers while preserving non-zero gradients and approximately 93% of the ideal deep-stack image-FLOP reduction.DPVR-PC retains 14 attention-mediated gradient paths but provides no compute saving, whereas DPVR-KV offers partial saving.

4. Experiments

Experiments evaluate DPVR on LLaVA-1.5-7B and 13B using eight multimodal benchmarks and split-sensitivity analyses. DPVR-LF achieves competitive accuracy with limited trainable parameters and reduced computation, while results remain sensitive to certain tasks and evaluation settings.

  • Experimental setup: Eight standard benchmarks evaluate DPVR-LF against baselines on LLaVA-1.5-7B and 13B.The setup uses LLaVA-1.5-7B and 13B with LLaVA-665k training data.
  • 7B main results: DPVR-LF matches or exceeds baselines on POPE, MME-Cognition, and ScienceQA, while staying within 0.5 pp of the best baseline on MMBench-EN and SEED.The weaker results occur on BLINK (−2.0 pp) and MMBench-CN (−1.9 pp).
  • 7B main results: 3% trainable parameters let DPVR-LF match or exceed full fine-tuning while reducing measured forward latency by −28.0%.The reported 7B headline uses a 3%-trainable side branch and compares against full fine-tuning.
  • 13B results: 13B DPVR-PC maintains below 0.3 pp 6-bench-mean variance across four split points, and DPVR-LF matches it on 6 of 8 metrics at s=28.The 13B results are single-seed, so standard deviations are not reported.
  • Split saturation: The split plateau spans s∈{20, 24, 28, 34} for 13B DPVR-LF with a 6-bench max−min of 0.23 pp, while s=12 lowers 7B accuracy by −1.0 pp.For 7B DPVR-LF, s∈[18,24] varies by Δ < 0.1 pp; the 13B plateau has two reported caveats.

4.5. Vision-Depth Ablation

The ablations show that a single-layer visual side branch and one fusion layer are sufficient across model sizes, while split points that are too shallow reduce accuracy. The selected configuration also yields substantial forward-latency savings, though the reported hardware profiles use specific protocols and split settings.

  • Vision-depth ablation: d_v=1 saturates 6-bench mean accuracy at 0.668 for 7B and 0.687 for 13B.Adding side-branch layers changes accuracy by at most −0.27 pp at 7B and within +0.02 to −0.12 pp at 13B.
  • Vision-depth ablation: BLINK at 7B drops from 0.407 to 0.394 as side-branch depth grows.The passage suggests that extra capacity may over-fit shallow layer-18 image representations before final fusion.
  • Split-layer ablation: s=18 and s=24 differ by only 0.05 pp, whereas s=12 drops −1.03 pp, driven mainly by a −3.5 pp POPE decline.The result supports splitting after visual representations have become sufficiently abstracted.
  • Fusion-layer ablation: Increasing fusion depth from K=1 to K=4 does not improve 6-bench accuracy, whose means remain within a 0.19 pp band.At 13B, K=1 versus K=2 changes the 6-bench mean by only −0.08 pp, so K=1 is retained with less compute.
  • Compute efficiency: DPVR-LF saves −26.8–−28.0% latency on A800 at 7B and −23.1% on 5880 Ada at 13B.The measurements cover multiple hardware platforms under a common batch, sequence-length, precision, and warmup protocol.

4.8. Prefill vs Decode Breakdown

DPVR-LF’s latency benefit occurs during prefill rather than decode under the current implementation. The saving remains positive across text lengths, although longer text dilutes the proportion of computation removed.

  • Decode: The current DPVR implementation falls back to a no-cache full-context forward at decode time.Different KV-cache shapes across shallow vision-bearing and deep text-only layers conflict with the vanilla decode loop’s uniform-shape expectation.
  • Practical scope: The reported prefill saving is the regime relevant to typical batched MLLM deployments using a single-pass forward over a long prompt.Decode-time KV-cache reuse is identified as future engineering work rather than an architectural change.
  • Text-length sensitivity: DPVR-LF’s saving remains positive across T_txt∈{64,128,256,512,1024}, peaking at −17.7% at T_txt=256 and softening to −8.0% at T_txt=1024.The passage attributes the shape to vision-token attention dominating at shorter lengths and text-quadratic attention diluting savings at longer lengths.

5. Discussion

The discussion argues that sparse late fusion can train effectively and that additional visual depth or fusion capacity is generally unnecessary. It also identifies capability, backbone-validation, and gradient-signal boundaries for the method.

  • Training dynamics: A single final-layer fusion provides enough signal to drive a 202M-parameter side branch to convergence.DPVR-LF reaches a final-50-step mean training loss of 1.64, while its 6-bench mean accuracy lies within 0.5 pp of DPVR-PC.
  • Fusion depth: K=2 matches K=1 in final loss at 1.63 versus 1.64 and stays within 0.18 pp in mean accuracy.This places a single fusion layer on the reported performance plateau.
  • Limitations: DPVR-LF loses the most ground on BLINK (−2.0 pp) and MMBench-CN (−1.9 pp).These results are presented as capability costs for multi-image relational reasoning and cross-lingual image-text alignment.
  • Limitations: Cross-backbone validation is restricted to a smoke test, with no DPVR-LF training yet reported on LLaVA-Next, Qwen-VL-2, or other newer backbones.Retuning the split layer and rerunning saturation analysis remain engineering follow-up.
  • Limitations: DPVR-LF has approximately 5% of DPVR-PC’s relative gradient density and requires a 2× learning rate to close the gap.The passage frames the weak gradient signal as a limitation of pushing toward fully text-only deep processing.

6. Conclusion

The paper identifies a mismatch between symmetric Transformer computation and asynchronous modality evolution, then proposes DPVR-LF to route saturated vision tokens separately. Across LLaVA-1.5-7B/13B, the method preserves benchmark performance while reducing computation, with one late fusion layer sufficient for perceptual competence.

  • Conclusion: Vision tokens saturate in middle layers while text tokens continue requiring full depth, exposing a structural mismatch in mainstream MLLMs.The conclusion describes deep visual updates as approaching a no-op.
  • Conclusion: DPVR-LF combines a one-layer side branch, a thirteen-layer text-only deep forward, and a single final-layer image-text fusion.This is the paper’s core modality-asymmetric routing design.
  • Conclusion: With only 3% trainable parameters, DPVR-LF matches or exceeds full fine-tuning across eight benchmarks while saving 25–30% of forward FLOPs.The conclusion also reports −28.0% measured A800 latency and a calibrated theoretical prediction of −26.8% at ρ=0.70.

Reproducibility

The paper releases code, configurations, evaluation scripts, and trained checkpoints to support reproducibility. It also releases figures and LaTeX source alongside raw per-task evaluation outputs.

  • Code, training configurations, evaluation scripts, and trained checkpoints are publicly available.
  • The experiments use a fixed transformers 4.57.1 environment with documented software and hardware details.
  • Raw evaluation outputs are committed as results.json files for each task and checkpoint.

A. DPVR-LF Training: Gradient-Sparsity Analysis

The gradient-sparsity analysis explains why a fully text-only deep stack cannot train the visual side branch under LLaVA labeling, and how final-layer fusion restores gradient flow. The loss supervises only assistant-response positions, so image representations require an explicit final attention path.

  • Gradient sparsity: Only assistant-response positions contribute to the cross-entropy loss because other positions receive the ignore label −100.
  • Gradient sparsity: DPVR-LF-ideal places image representations directly from the side branch while processing deep text positions through a fully text-only stack.
  • Gradient sparsity: DPVR-LF-ideal receives no gradient signal at the side branch under the standard cross-entropy loss.
  • Gradient recovery: Final-layer fusion reassembles image representations before attention, creating a non-zero gradient path into the trainable side branch.

B. Implementation Details

The implementation uses fixed training and evaluation software, the LLaVA-1.5 665k mixture, and standard benchmark splits. It includes separate implementation files for DPVR baselines and DPVR-LF, with reproducibility tables for three-seed runs.

  • Software stack: Training and evaluation use PyTorch 2.9.0+cu128, transformers 4.57.1, trl 0.23.0, accelerate 1.11.0, and bitsandbytes 0.49.2.
  • Dataset: Training uses 665k LLaVA-1.5 multimodal samples for one epoch without further filtering or subsampling.
  • Implementation and evaluation: The repository separates DPVR-PC, DPVR-KV, and DPVR-LF implementations, while Tables 10 and 11 report three-seed results.

C. Per-Seed Breakdown and Shared-Shuffle Caveat

The reported seed variation reflects initialization randomness under a fixed minibatch order rather than full data-order randomness. The authors identify this as a limitation and leave rerunning with corrected shuffle seeding for camera-ready work.

  • Shared-shuffle caveat: Reported standard deviations characterize different weight initializations along identical data trajectories.
  • Variance interpretation: 0.001–0.011 absolute standard deviations on 7B three-seed runs capture initialization variance but omit data-order variance.
  • Planned correction: A one-line seeding fix would measure full data-order variance, which the authors expect to widen intervals modestly without changing headline conclusions.
Loading 2606.09131v1…