PaperlayerView exact revision on arXiv

Source-linked AI summary

Late-Layer Fusion is Enough: Dual-Path Vision Token Routing for Multimodal Large Language Models under Visual Saturation

Siyuan Liu, Jinyang Wu

arXiv:2606.09131v1cs.AIcs.CLcs.CVcs.LG

TL;DR

MLLMs often process image and text tokens uniformly even though vision tokens saturate earlier while text tokens continue to benefit from deeper computation. This paper introduces DPVR-LF, routing vision tokens through a side branch and fusing them with text at the final layer; on LLaVA-1.5-7B and 13B, it matches or exceeds full fine-tuning across eight benchmarks while saving 25–30% of forward FLOPs.

  • Problem

    MLLMs commonly process image and text tokens uniformly despite vision tokens saturating earlier while text tokens continue to depend on deeper computation.

    Source: §1 · p. 2

  • Method

    DPVR-LF branches vision tokens into a trainable side path while text continues through the deep stack, then fuses the streams in a single final-layer block.

    Source: §1 · p. 2

  • Results

    25–30% of forward FLOPs saved while DPVR-LF matches or exceeds full fine-tuning across eight standard benchmarks on LLaVA-1.5-7B and 13B.

    Source: §1 · p. 2

  • Takeaways & Limitations

    In LLaVA-1.5-7B, omitted deep image updates leave the final representation effectively unchanged, supporting the conclusion that those updates are essentially redundant.

    Source: §5.1 · p. 13

    DPVR-LF loses 2.0 pp on BLINK and 1.9 pp on MMBench-CN, indicating costs for multi-image relational reasoning and cross-lingual image-text alignment.

    Source: §5.3 · p. 14

Abstract

from arXiv · show

Multimodal large language models (MLLMs) commonly inherit the deep, symmetric Transformer backbone designed for unimodal text modeling, and apply the same computation uniformly to image and language tokens. This design overlooks a key modality asymmetry: image and text tokens differ substantially in information density, redundancy, and required reasoning depth. Through a layer-wise analysis of LLaVA-1.5, we observe that vision tokens tend to saturate in the middle layers. Specifically, text-to-image attention decreases from 0.68 at layer 0 to 0.07 by layer 4, and stabilizes near 0.04 after layer 18, whereas text tokens continue to benefit from deep semantic processing. These findings suggest a mismatch between architectural symmetry and depth-asynchronous modality evolution, resulting in redundant visual computation and possible drift in perceptual representations during deep task-specific adaptation. Motivated by this, we propose Dual-Path Vision Token Routing (DPVR), a modality-asymmetric routing framework for efficient MLLMs. Its core instantiation, DPVR-LF (Late-Layer Fusion), routes vision tokens at the saturation point into a one-layer trainable side branch, runs a thirteen-layer text-only forward that skips image positions in the deep stack, and re-fuses the visual and textual streams only at the final layer. With approximately 3% trainable parameters, DPVR-LF preserves competitive multimodal performance on standard benchmarks while reducing visual computation in the deep Transformer stack. The results challenge the conventional assumption that vision tokens must traverse all deep language-model layers, and indicate that a single late fusion layer can be sufficient for maintaining strong perceptual competence in LLaVA-style MLLMs.

1. Introduction

LLaVA-1.5 analysis indicates that vision representations saturate before text representations, motivating DPVR-LF’s shallow visual route and final-layer fusion. The paper reports competitive benchmark performance with fewer trainable parameters and reduced computation.

Vision saturation analysis

  • Vision-token adjacent-layer cosine similarity stays above 0.92, while text-to-image attention falls from 0.68 at L0 to 0.07 at L4 and stabilizes near 0.04.Vision tokens reach the prediction space at L22, one layer before text tokens at L23.Source: §1 · p. 2

Motivation

  • These observations suggest visual representations saturate before the final layers, while text continues to rely on deeper processing for semantic composition and response generation.The paper argues that deep visual updates may be redundant and may perturb perceptual information captured earlier during adaptation.Source: §1 · p. 2

DPVR-LF

  • DPVR-LF branches vision tokens at s=18 for 7B or s=28 for 13B into one trainable layer, skips image positions in the deep text-only forward, and fuses streams at the final layer.The final full-attention block preserves a trainable image-to-text gradient path while avoiding repeated deep updates to image positions.Source: §1 · pp. 2–3

Contributions

  • With 3% trainable parameters, DPVR-LF matches or exceeds full fine-tuning across eight benchmarks while saving 25–30% of forward FLOPs.The reported A800 latency saving is −28.0%; the calibrated theoretical prediction is −26.8% at ρ=0.70.Source: §1 · pp. 2–3

Robustness and ablations

  • The split sweep finds a robust s∈[18, 24] plateau, with six-benchmark mean varying within 0.1 pp; a second fusion layer adds only +0.18 pp, within run-to-run noise.The 𝑑𝑣=1 vision-depth saturation point also replicates at 13B scale.Source: §1 · p. 3

Mechanism and efficiency

  • The lone fusion layer concentrates text-to-image attention at 1.77× the vanilla baseline, and latency savings remain positive across a 16× text-token-length sweep.The shared shallow stack is bit-identical to vanilla LLaVA (median cosine >0.9998 on 500 samples at both scales); prefill latency saving is −23.4% on 13B / 5880 Ada and reproduces across three hardware platforms.Source: §1 · p. 3

2. Related Work

Prior work spans multimodal model designs, visual-token reduction, conditional computation, parameter-efficient tuning, and layer-wise analysis. DPVR is positioned as a structural, modality-conditional change to the decoder-only backbone that preserves visual tokens while redirecting their deep computation.

Multimodal large language models

  • MLLMs evolved from contrastive image-text alignment and frozen-encoder bridges to projector-connected decoder-only models, whose shared deep backbone motivates DPVR.The third generation processes mixed-modality sequences with a single decoder-only LLM and has become dominant.Source: §2 · p. 3

Token reduction in MLLMs

  • Unlike visual-token pruning or merging, DPVR preserves all 576 visual tokens and changes their computation graph so they skip 13 of 14 deep Transformer layers.Prior MLLM token-reduction methods reduce the number of visual tokens, while ToMe and EViT operate inside the vision encoder before the projector.Source: §2 · p. 3

Architectural compute reduction and conditional compute

  • DPVR uses deterministic modality-conditional routing and a fixed 13-layer text-only segment, unlike stochastic layer dropping, sparse experts, weight pruning, or model-level query routing.The fixed compute graph is designed to be measurable and deployable; token routing remains within one model.Source: §2 · p. 4

Cross-modal architectures and decoder-only design

  • DPVR restores structural separation within decoder-only LLaVA using a one-layer side branch and one fusion layer, without new pretraining or changes to the projector or vision encoder.Its weight space remains compatible with off-the-shelf LLaVA checkpoints and instruction-tuning data.Source: §2 · p. 4

Parameter-efficient fine-tuning

  • DPVR concentrates a 202M-parameter full-capacity transformer block, 3% of the 7B backbone, at the routing junction rather than spreading low-rank updates across 32 layers.The paper compares DPVR-LF with LoRA r=16 and r=64 in §4.2.Source: §2 · p. 4

Cross-layer analysis of language and vision-language models

  • Prior layer-wise studies did not jointly use adjacent-layer cosine similarity, text-to-image attention mass, and logit-lens KL divergence to localize visual saturation in an MLLM.The paper uses the resulting saturation layer, L_sat ≈ L_trans−4, to set DPVR-LF’s split point.Source: §2 · p. 4

Broader directions in LLM reasoning and retrieval

  • DPVR is orthogonal to work on reasoning guidance, retrieval, and knowledge distillation because it restructures where computation flows inside an existing backbone.The paper leaves combinations such as distilling a DPVR-LF-trained student from a full-attention teacher for future work; reasoning and retrieval lines instead address what the LLM reasons over or how it is guided.Source: §2 · pp. 4–5

Visual instruction tuning

  • LLaVA-665k is shared across DPVR variants, LoRA ranks, and the cited FullFT row in Table 2 to keep comparisons with prior baselines on the same training data.Other visual instruction-tuning corpora mentioned include ShareGPT4V and LLaVA-Next, which expands to approximately 1.4M examples.Source: §2 · p. 5

3. Methodology

The methodology uses layer-wise evidence of modality-asynchronous saturation to motivate DPVR-LF, which processes vision through a trainable side branch and fuses it with text only at the final layer. It also defines routing baselines and a shared training procedure for evaluating the approach.

3.1. Visual Saturation Insight

  • Vision-token adjacent-layer cosine similarity is at least 0.92 from L0, while text-to-image attention drops 10× in four layers and asymptotes to 0.04 after L18.Figure 2 reports 50%-transition layers of L22 for vision and L23 for text; the analysis uses 500 LLaVA-665k samples and three complementary statistics.Source: Figure 2 · p. 5, §3.1 · p. 5
  • The 13B vision 50%-transition shifts from L22 (69% of depth) at 7B to L33 (82%) at 13B, while the text transition shifts from L23 to L26.This asynchronous scaling motivates a 13B split-layer sweep over s∈{20, 24, 28, 34}; at 7B, 6-bench mean accuracy varies by less than 0.1 pp over s∈[18, 24].Source: §3.1 · pp. 5–6

3.2. DPVR-LF: Late-Layer Fusion

  • DPVR-LF routes image tokens at layer 18 through a one-layer trainable side branch, skips image positions in 13 deep layers, and re-fuses image and text at the final layer.This removes image computation from 13 of 14 deep-stack layers while retaining a gradient path through final-layer attention.Source: §3.2 · pp. 6, 8
  • Without final-layer fusion, DPVR-LF-ideal is untrainable because image-position logits receive zero loss gradient; final-layer text-to-image attention restores gradient flow to the side branch.The single fusion path provides roughly 5% of DPVR-PC’s gradient density, compensated by a 2× learning rate (1e−4 vs 5e−5).Source: §3.2 · p. 7
  • At L39, head-mean text-to-image attention mass is 0.388 for DPVR-LF s=24 13B versus 0.219 for vanilla LLaVA-1.5-13B, a 1.77× increase.The highest-attending head is similar at approximately 0.80, so the increase reflects broader attention across heads.Source: §3.2 · p. 8

3.3. Baselines

  • DPVR-PC retains full deep-layer image processing and 14 gradient paths, whereas DPVR-KV skips image queries, output projections, and FFNs for an approximately 22% theoretical forward saving.All three variants use the same 202M single-transformer trainable budget, so accuracy gaps reflect routing structure rather than capacity.Source: §3.3 · p. 8

3.4. Training Procedure

  • All variants freeze everything except the side-branch transformer and train for one epoch on LLaVA-665k; trainable budgets are 202M (3.04%) at 7B and 313M (2.4%) at 13B.DPVR-PC and DPVR-KV use LR 5e−5, while DPVR-LF uses 1e−4; other hyperparameters are held constant.Source: §3.4 · p. 8

4. Experiments … 4.4. Split Saturation Curve

DPVR-LF achieves competitive 7B and 13B benchmark performance with a stable split-layer plateau across model sizes and variants. The experiments evaluate these methods against standard baselines and report specific task-level and split-point caveats.

4.1. Setup

  • Experiments use LLaVA-1.5-7B and 13B with LLaVA-665k, evaluating eight standard benchmarks against vanilla, fine-tuning, LoRA, FastV, and DPVR baselines.Training uses one A800 80GB; evaluation uses RTX 5880 Ada 48GB cards. The benchmarks include POPE, MME-P/C, MMBench-EN/CN, ScienceQA, SEED-Bench, and BLINK.Source: §4.1 · pp. 8–9

4.2. Main Results on LLaVA-1.5-7B

  • 3%-trainable DPVR-LF matches or exceeds 7B full fine-tuning while reducing measured forward latency by −28.0%.Across eight benchmarks, it matches or exceeds all baselines on POPE, MME-Cognition, and ScienceQA, and is within 0.5 pp of the best baseline on MMBench-EN and SEED.Source: §4.2 · p. 9, Table 2 · p. 9

4.3. Main Results on LLaVA-1.5-13B

  • DPVR-LF matches DPVR-PC on 6 of 8 metrics at 13B, while DPVR-PC’s 6-bench mean varies by less than 0.3 pp across four split points.The DPVR-LF main row uses split s=28; the reported 13B results are single-seed.Source: §4.3 · p. 9, Table 3 · p. 10

4.4. Split Saturation Curve

  • Across model sizes and variants, the 6-bench mean varies by less than 0.3 pp across broad split-layer plateaus.The plateaus are s∈[18, 24] for 7B DPVR-LF and s∈[20, 34] for both 13B DPVR-PC and DPVR-LF; Figure 4 also shows that 7B DPVR-LF at s=12 drops by −1.0 pp.Source: §4.4 · p. 9, Figure 4 · p. 10
  • MME-Cognition spans 277–313 across the 13B DPVR-LF endpoints, while POPE at s=28 is 0.8543 versus at least 0.859 at the other endpoints.The paper describes the POPE difference as a single-seed artefact or hyperparameter-specific ripple; the remaining plateau is unperturbed.Source: §4.4 · p. 10

4.5. Vision-Depth Ablation · 4.6. DPVR-LF Ablation · 4.7. Compute Efficiency

Across the ablations, a one-layer vision side branch and a single fusion layer sit on accuracy plateaus, while DPVR-LF reduces forward latency across the reported hardware and model sizes. The sections also show that DPVR-PC adds latency rather than reducing forward FLOPs.

4.5. Vision-Depth Ablation

  • At d_v=1, 6-bench mean accuracy is 0.668 at 7B and 0.687 at 13B, with deeper branches changing the means by at most 0.27 pp and 0.12 pp, respectively.The ablation identifies d_v=1 as the cross-size saturation point; adding side-branch capacity is neutral at best and mildly harmful at worst.Source: §4.5 · p. 10, Table 4 · p. 11
  • At 7B, BLINK falls from 0.407 to 0.394 as d_v grows, the only monotone signal in the vision-depth ablation.The passage suggests deeper side branches may overfit shallow layer-18 image hidden states.Source: §4.5 · p. 10, Table 4 · p. 11

4.6. DPVR-LF Ablation

  • At split layer s=18 versus s=24, the 6-bench means differ by only 0.05 pp, while s=12 drops −1.03 pp, driven mainly by a −3.5 pp POPE decline.The result indicates that splitting too shallow leaves the side branch with insufficiently abstracted image hidden states.Source: §4.6 · p. 11, Table 5 · p. 11
  • At 7B and s=18, the K sweep stays within a 0.19 pp 6-bench band (0.6631–0.6650), and increasing K from 1 to 4 does not improve accuracy.The N=3 variance at K=2 and K=3 confirms the plateau is not a single-seed artefact; K=1 is retained for lower compute, and at 13B, K=1 to K=2 changes the mean by only −0.08 pp.Source: §4.6 · p. 11, Table 5 · p. 11

4.7. Compute Efficiency

  • Latency was measured on A800, RTX PRO 6000 Blackwell, and RTX 5880 Ada with B=1, 576 image tokens, 128 text tokens, bf16, 10 warmups, and 100 measured forwards, except the marked 7B A800 B=4 prefill-only measurement.For 13B, latency was profiled at endpoint splits s=24 and s=34 rather than the accuracy-main split s=28; those endpoints bracket the reported saving.Source: §4.7 · p. 12, Table 6 · p. 12
  • DPVR-LF cuts forward latency by −26.8–−28.0% on 7B A800, −14.8% on 13B Blackwell at s=24, and −23.1% on 13B 5880 Ada.The reported savings are consistent across hardware and widen on the more compute-bound 5880 Ada.Source: §4.7 · p. 12, Table 6 · p. 12
  • DPVR-PC adds +6.32% latency at 7B and +4.87% at 13B on Blackwell, confirming that its contribution is trainable-parameter reduction rather than forward-FLOPs savings.The passage reports +5–+6% latency at both sizes overall.Source: §4.7 · p. 12, Table 6 · p. 12

4.8. Prefill vs Decode Breakdown · 4.9. Sensitivity to Text-Token Length 𝑇txt

DPVR-LF’s latency savings accrue at prefill, while its current decode implementation lacks KV-cache reuse; its savings remain positive across a 16× sweep of text-token lengths. The text-length results peak at 256 tokens and weaken as text attention costs grow.

4.8. Prefill vs Decode Breakdown

  • −23.4% prefill reduction at 𝑠= 24 on 5880 Ada nearly matches the −23.1% forward-total reduction.The prefill and total-forward reductions are reported for DPVR-LF; Table 7 measures prefill and per-token decode latency on 13B / 5880 Ada with 𝑇img = 576 and 𝑇txt = 128.Source: §4.8 · p. 12, Table 7 · p. 13
  • At decode time, the current DPVR implementation falls back to a no-cache, full-context forward at every step because shallow and deep layers have different KV-cache shapes.The vanilla LlavaForConditionalGeneration decode loop expects a uniform shape.Source: §4.8 · p. 12
  • A TD-aware decode loop could re-fuse vision KV at the fusion layer and recover decode-time KV-cache reuse; the authors flag this as future engineering work.The current decode limitation is described as an implementation limitation, not an architectural one.Source: §4.8 · p. 12
  • The reported prefill-time saving stands on its own for typical batched MLLM deployments, which use a single-pass forward over a long prompt.Table 7 distinguishes single-pass prefill from per-token decode during a 64-token continuation.Source: §4.8 · p. 12, Table 7 · p. 13

4.9. Sensitivity to Text-Token Length 𝑇txt

  • DPVR-LF savings remain positive across the 16× 𝑇txt sweep, peaking at −17.7% at 𝑇txt = 256 and softening to −8.0% at 𝑇txt = 1024.The sweep covers 𝑇txt ∈{64, 128, 256, 512, 1024} on Blackwell; Table 8 reports method latency changes against vanilla 13B.Source: §4.9 · p. 13, Table 8 · p. 13
  • DPVR-PC overhead decreases monotonically from +4.88% to +3.16% as text length grows, because the relative cost of its parallel deep-image loop shrinks.These overhead values are reported across the text-length sweep.Source: §4.9 · p. 13
  • At short text lengths, vision-token attention dominates forward cost, while growing text-quadratic attention dilutes the fraction of compute DPVR-LF skips at long lengths.The typical 128–512 instruction-tuning window is where DPVR-LF savings peak.Source: §4.9 · p. 13

5. Discussion

DPVR-LF’s single late fusion preserves downstream visual information and can train a large side branch despite sparse gradient paths, but deeper fusion remains valuable for some tasks. The discussion identifies transfer and optimization limits alongside directions for extending the approach.

5.1. Why DPVR-LF works

  • By L22, vanilla LLaVA-1.5-7B vision tokens have completed their prediction-space transition, making a single final-layer image-text fusion sufficient for accurate responses.The omitted 13 deep-layer image attentions are described as redundant: removing them leaves the final representation effectively unchanged, consistent with deep vision-token updates approaching a no-op.Source: §5.1 · p. 13

5.2. Training dynamics with sparse gradient paths

  • DPVR-LF reaches a final-50-step mean loss of 1.64 and keeps 6-bench mean accuracy within 0.5 pp of DPVR-PC using a 2× learning rate.The side branch receives one attention-mediated gradient path rather than DPVR-PC’s 14; the higher learning rate (1e−4 vs 5e−5) compensates, and loss and accuracy converge in the same ballpark.Source: §5.2 · p. 13
  • With K=2, final loss is 1.63 versus 1.64 for K=1, and mean accuracy differs by within 0.18 pp on the N=3 mean, indicating a single-fusion plateau.The passage describes this as an isoquant plateau, while noting the task trade-off motivates task-conditioned fusion.Source: §5.2 · p. 14, §5.4 · p. 14

5.3. Limitations

  • DPVR-LF loses 2.0 pp on BLINK and 1.9 pp on MMBench-CN, while LLaVA-Next validation remains a smoke test and newer backbones remain untrained.The losses indicate a capability cost for multi-image relational reasoning and cross-lingual image-text alignment. LLaVA-Next loaded and generated coherent held-out captions, but the split layer still needs retuning and saturation analysis; DPVR-LF has not been trained on it, Qwen-VL-2, or other newer backbones. Gradient density is approximately 5% of DPVR-PC’s and requires a 2× learning rate; DPVR-LF-ideal with no fusion is strictly untrainable.Source: §5.3 · p. 14

5.4. Future Work

  • Future work proposes extending DPVR to SigLIP and LLaVA-Next, testing selective or task-conditioned fusion, adjusting side-branch learning-rate schedules, and combining it with token pruning or distillation.The selective-fusion direction includes sparse non-contiguous layers between K=1 and DPVR-LF-ideal; task-conditioned fusion would allocate more capacity to multi-image and cross-lingual inputs.Source: §5.4 · p. 14

6. Conclusion

The conclusion presents DPVR-LF as a response to the mismatch between vision-token saturation and text tokens’ need for deep processing. It reports competitive benchmark performance with reduced computation and argues that a single final fusion layer is sufficient for perceptual competence.

6. Conclusion

  • With only 3% trainable parameters, DPVR-LF matches or exceeds full fine-tuning across eight benchmarks on LLaVA-1.5-7B/13B while saving 25–30% of forward FLOPs.The conclusion also reports −28.0% measured A800 latency and a calibrated theoretical prediction of −26.8% at 𝜌= 0.70.Source: §6 · p. 14
  • DPVR-LF combines a one-layer side-branch single transformer, a thirteen-layer text-only deep forward, and a single final-layer image-text fusion.The conclusion frames the final fusion layer as sufficient for perceptual competence in modern MLLMs.Source: §6 · p. 14
  • The paper identifies a structural mismatch: vision tokens saturate in the middle layers while text tokens still require the full depth.Deep updates to vision tokens approach a no-op.Source: §6 · p. 14

Reproducibility

The authors release the code, experiment artifacts, and raw evaluation outputs to support reproduction and direct audit. They also document a common software environment and retain per-seed training records.

Reproducibility

  • Code, training configurations, evaluation scripts, trained checkpoints, figures, and the paper’s LaTeX source are available in the public repository.https://github.com/Inner-Magma/Dual-Path-Token-RoutingSource: Reproducibility · p. 14
  • All experiments use transformers 4.57.1, while raw per-task evaluation outputs, per-seed training trajectories, run logs, and trainer_state.json files are retained for audit.Appendix B provides full software and hardware details and the training-time hyperparameter table; the training records are referenced in Table 5.Source: Reproducibility · p. 14

A. DPVR-LF Training: Gradient-Sparsity Analysis

Under LLaVA’s response-only labeling, DPVR-LF-ideal gives its image side branch no gradient signal, while DPVR-LF’s final-layer fusion restores a gradient path to that branch.

A. DPVR-LF Training: Gradient-Sparsity Analysis

  • Only assistant-response tokens have non-−100 labels, so the cross-entropy loss with ignore_index = -100 is restricted to response positions.The passage defines the response-position set as R.Source: A. DPVR-LF Training: Gradient-Sparsity… · p. 16
  • DPVR-LF-ideal receives no gradient signal at its image side branch because assistant-response positions are text positions and the loss produces no gradient at other positions.The final hidden state takes image positions directly from the side branch, while the response-position set is disjoint from image positions.Source: A. DPVR-LF Training: Gradient-Sparsity… · pp. 16–17
  • DPVR-LF recovers gradient flow by reassembling image features before final-layer attention, letting text queries attend to image keys and backpropagate through one attention path to the side branch.The resulting non-zero gradient with respect to out_image propagates into the single-layer branch parameters.Source: A. DPVR-LF Training: Gradient-Sparsity… · p. 17

B. Implementation Details

The implementation specifies the software and evaluation stack, training data and seed behavior, and the planned release of model-specific code. Its seed configuration affects initialization and dropout randomness but not data-loader shuffling.

Software stack

  • Runs use PyTorch 2.9.0+cu128, transformers 4.57.1, trl 0.23.0, accelerate 1.11.0, bitsandbytes 0.49.2, Python 3.10, and bf16; evaluation uses lmms-eval 0.5.0 with llava_hf and use_cache=False.The cache setting is mandatory for the TD prefill path; Appendix C discusses a related data-loader-seed caveat.Source: B. Implementation Details · p. 17

Dataset

  • Training uses one epoch over 665k LLaVA-1.5 visual instruction-tuning samples at effective batch size 4, without further filtering or subsampling.All eight evaluation benchmarks use standard splits, with metric handler keys recorded in the released evaluation chain configurations.Source: B. Implementation Details · p. 17

Random seeds

  • Three training seeds (1, 2, 3) supply the N=3 cells in Table 2 and K=2 / K=3 rows in Table 5, but seed-driven variance is initialization-only because the seed does not control data-loader shuffling.Single-seed runs use seed 42.Source: B. Implementation Details · p. 17

Code

  • The source and training scripts are planned for release upon paper acceptance; listed model files implement the DPVR-PC, DPVR-KV, and DPVR-LF variants.The files are token_diversion.py, token_diversion_substitution.py, and token_diversion_x3_fusion.py, respectively.Source: B. Implementation Details · p. 17

C. Per-Seed Breakdown and Shared-Shuffle Caveat

The reported seed standard deviations capture initialization variance under a shared minibatch order, not full data-order variance. The authors identify a reseeding fix and expect randomized shuffling to widen intervals modestly without changing headline conclusions.

C. Per-Seed Breakdown and Shared-Shuffle Caveat

  • Table 2 and Table 5 seed standard deviations reflect initialization-driven convergence under identical LLaVA-665k minibatch order; per-seed numbers appear in Tables 10 and 11.The seed parameter controls weight initialization and dropout but not the HuggingFace Trainer’s data-loader shuffle RNG.Source: C. Per-Seed Breakdown and… · p. 18
  • 0.001–0.011 absolute standard deviations on 7B 3-seed runs and BLINK ±.005 for DPVR-LF and ±.006 for DPVR-PC are presented as a meaningful lower bound on full seed variance.The reported values capture initialization variance; randomized data shuffling could widen the intervals.Source: C. Per-Seed Breakdown and… · p. 18
  • Calling transformers.set_seed(cfg.seed) immediately before Trainer.train() would reseed Python, NumPy, torch, and the Trainer shuffle RNG; the authors expect modestly wider intervals without changing headline conclusions.A Table 2 rerun under the fix would report full data-order variance; the headline conclusions depend on means and per-row orderings rather than standard-deviation magnitude.Source: C. Per-Seed Breakdown and… · p. 18

Related papers

Loading 2606.09131v1…