PaperlayerView exact revision on arXiv

Source-linked AI summary

Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs

Donghyeon Cho, Joowon Kim, Seungho Shin, Eunho Yang

arXiv:2610.36671v1cs.CVcs.AIcs.LG

TL;DR

DVLMs process visual tokens at every denoising step even though image access often does not change the prediction, and existing efficiency methods do not decide whether an individual step needs the image. LOWN uses agreement between text-only and image-conditioned predictions to route image access and set how many tokens to unmask. Across ten benchmarks, it retains 99.6% of LLaDA-V’s score at 0.35× the decoding FLOPs.

  • Problem

    DVLMs repeatedly process visual tokens that make up 89–97% of the input, while existing efficiency methods do not ask whether the current decoding step needs the image.

    Source: §1 · p. 1

  • Method

    LOWN runs a TEXT-ONLY forward at each step, reuses the latest FULL prediction when they agree, and runs a new FULL forward when they disagree.

    Source: §1 · p. 1

  • Results

    99.6% of LLaDA-V’s score is retained across ten benchmarks at 0.35× the decoding FLOPs.

    Source: §6 · p. 10

  • Takeaways & Limitations

    LOWN’s image-access routing can be combined with visual token pruning or key-value caching for further efficiency gains.

    Source: §1 · p. 2

    The number of FULL forwards, and therefore the cost, cannot be set before decoding; combining LOWN with visual token pruning gives some cost control through its retention ratio.

    Source: §6 · p. 10

Abstract

from arXiv · show

Diffusion vision-language models (DVLMs) process their entire input at every denoising step. Since visual tokens make up 89-97% of this input on the benchmarks we study, most of the decoding computation is spent repeatedly processing the same image. Visual token pruning methods partially relieve this burden by removing part of the visual tokens, but they lose the removed information and still process the remaining image at every step. To measure how often the image is actually needed, we compare LLaDA-V's predictions with and without the image at every decoding step and find that removing the image changes a token being unmasked at only 22.8% of steps on average and at as few as 7.9%. We therefore propose LOWN, a decoding method that decides at every step whether to process the image and how many tokens to unmask. LOWN predicts the masked tokens without the image and processes the image only when this prediction disagrees with the last image-conditioned prediction. Agreement certifies the earlier prediction for reuse, and the number of certified tokens sets how many are unmasked in that step. We further observe that the need for the image differs across benchmarks, so LOWN's adaptive decision is needed rather than a fixed schedule. Without training or a tuned threshold, LOWN retains 99.6% of LLaDA-V's score on ten benchmarks at 35.3% of the decoding FLOPs on average. LOWN also generalizes across LLaDA-based DVLMs, composes orthogonally with visual token pruning, and runs under key-value caching for further efficiency gains. Code and project page are available at https://hyeon9698.github.io/LOWN/

1. Introduction

DVLMs repeatedly process visual tokens that dominate their input, even though image access often does not change the prediction. LOWN uses prediction agreement to decide when to process the image and how many tokens to unmask, retaining nearly all baseline quality at substantially lower decoding cost.

1. Introduction

  • Visual tokens account for 89–97% of LLaDA-V’s input, and a forward without them costs 3–10% as much as a FULL forward.Standard DVLM inference processes the full multimodal sequence at every step, repeatedly recomputing the unchanged visual input.Source: §1 · p. 1
  • Existing pruning, caching, and unmasking methods reduce forward cost or decoding steps, but do not test whether the current step needs the image.The paper frames this as deciding whether removing the image would change the model’s decision at the current decoding state.Source: §1 · p. 1
  • FULL and TEXT-ONLY predictions agree at 83.4% of positions selected for unmasking, while image-dependent steps range from 7.9% on MMBench to 43.8% on DocVQA.Image-dependent steps tend to cluster early within trajectories, and their overall share varies across benchmarks, making fixed schedules a poor fit.Source: §1 · p. 1
  • LOWN reuses the latest image-conditioned prediction when a TEXT-ONLY forward agrees and runs a new FULL forward when they disagree.The confidence-ordered agreements also determine how many tokens to unmask; the method requires neither training nor a task-specific threshold.Source: §1 · pp. 1–2
  • 99.6% of LLaDA-V’s baseline score is retained at 0.35× the decoding FLOPs across ten benchmarks.LOWN scores highest among accelerated methods on eight of ten benchmarks; it also composes with pruning and key-value caching.Source: Figure 1 · p. 2, §1 · p. 2

2. Related Work

Prior efficiency methods reduce visual tokens, reuse computation, or unmask more tokens per step. LOWN instead uses agreement between text-only and image-conditioned predictions to decide both image access and unmasking, while retaining the predictions needed to compose with pruning and caching.

2. Related Work

3. Background and Motivation

DVLMs iteratively recover masked response tokens while conditioning on fixed visual and prompt tokens. Measurements along LLaDA-V trajectories show substantial agreement between image-conditioned and text-only predictions, but image dependence varies over steps and benchmarks, motivating adaptive image access.

3.1. Preliminaries

  • A DVLM starts with a fully masked response and predicts response tokens over denoising steps, conditioning on fixed visual and prompt tokens.At each step, the response has masked positions; LLaDA-V uses a FULL forward and unmasks the most confident positions, with k equal to response length divided by T.Source: §3.1 · p. 3

3.2. When Does a DVLM Need the Image?

  • 83.4% of positions selected for unmasking receive the same top-1 token from FULL and TEXT-ONLY forwards, averaged over ten benchmarks.The FULL forward determines the trajectory; TEXT-ONLY outputs are recorded without affecting its decisions.Source: §3.2 · p. 3
  • 85.0% of final tokens had already appeared as top-1 predictions at the same position before unmasking, although the baseline kept recomputing those positions with the image.Source: §3.2 · p. 4
  • Image-dependent steps range from 7.9% on MMBench to 43.8% on DocVQA and average 22.8%; disagreement is greatest early on eight of ten benchmarks.The variation across trajectories and tasks makes fixed image-access schedules poorly matched; Figure 2 maps disagreement over token position and denoising step and reports per-benchmark step shares.Source: §3.2 · p. 4, Figure 2 · p. 4
  • A TEXT-ONLY forward costs 3–10% of a FULL forward, making it a low-cost per-step test for deciding whether to process the image.The next section uses agreement between the two forwards as that test.Source: §3.2 · p. 4

4. Method

LOWN separates image-access decisions from token-unmasking decisions by testing the latest image-conditioned anchor with a TEXT-ONLY forward. Certificates determine whether a FULL forward is needed and how far decoding advances.

4. Method

  • LOWN tests whether the response state needs the image and how far decoding can advance, without changing model parameters or the attention pattern.It retains the latest image-conditioned prediction as an anchor and tests it with a TEXT-ONLY forward.Source: §4 · p. 4

4.1. Anchor and Certificate

  • Candidates are checked in decreasing anchor-confidence order; the longest confidence-ordered run of agreements forms the certified prefix, and each FULL forward refreshes the anchor.The anchor stores predictions and confidences for positions that remain masked.Source: §4.1 · pp. 4–5

4.2. Certificate-Guided Decoding

  • If the certified prefix reaches k, LOWN skips FULL and unmasks the whole prefix; otherwise it refreshes the anchor and unmasks k positions, disagreements first.Both branches unmask at least k tokens, so LOWN takes no more than the baseline's T steps.Source: §4.2 · p. 5

4.3. Cost Analysis

  • Relative FLOPs reflect FULL-forward frequency and TEXT-ONLY step frequency and unit cost; visual pruning lowers FULL-forward cost, while LOWN lowers how often FULL runs.The method's cost model defines the TEXT-ONLY unit cost relative to a FULL forward; pruning and LOWN can be composed.Source: §4.3 · pp. 5–6

5. Experiments

Experiments evaluate LOWN across ten benchmarks and test its composition with pruning and caching, transfer to other models, and image-access decisions. LOWN retains nearly all baseline score at substantially lower decoding cost, while analyses compare adaptive routing with fixed or random schedules and examine certificate reliability.

5.1. Implementation Details

  • Evaluation covers ten benchmarks on LLaDA-V and LaViDa, plus EgoSchema and MLVU on VidLaDA-8B; quality uses task-specific metrics and FLOPs are relative to each backbone.Comparators receive at least LOWN’s FLOPs budget, and standalone LLaDA-V decoding uses batch size 1 without key-value caching.Source: §5.1 · pp. 6–7

5.2. Main Results

  • LOWN retains 99.6% of LLaDA-V’s baseline score on average at 0.353× baseline FLOPs and scores highest among accelerated methods on eight of ten benchmarks.Capping each benchmark at 100% gives 98.6%, since four benchmarks exceed the baseline.Source: §5.2 · p. 7, Table 1 · p. 7
  • With RedVTP at 50% retention, LOWN reaches normalized average 98.3 at 0.223× baseline FLOPs and scores highest on all ten benchmarks.On MathVista, RedVTP at 50% retention with LOWN scores 54.9 at 0.195× FLOPs, versus 51.7 at 0.354× for RedVTP at 30% retention.Source: §5.2 · pp. 7–8, Table 1 · p. 7
  • Adding LOWN to Fast-dLLM reduces mean generation time by 38.1–50.1% and FLOPs by 44.1–58.9%, but scores remain below Fast-dLLM on all three tasks.Combined scores are within 2.0 points of LLaDA-V, with a 5.0-point decrease on LLaVA-Bench versus Fast-dLLM.Source: §5.2 · p. 8, Table 2 · p. 8
  • LOWN transfers to LaViDa at 99.7% of baseline score and 0.174× FLOPs, and to VidLaDA-8B at 0.255×/0.272× FLOPs on EgoSchema/MLVU, scoring 60.2/43.3 versus 60.6/41.8.LaViDa’s Figure 5 identifies the baseline and pruning/composition markers.Source: §5.2 · pp. 8–9, Figure 5 · p. 8, Table 3 · p. 8
  • Across ten LLaDA-V benchmarks, standalone LOWN reduces generation time by 57.4% on average and up to 73.2%.Timing uses per-example generation time; the reported mean averages all per-example times from one full evaluation-set execution.Source: §5.2 · p. 9

5.3. Ablation and Analysis

  • Against matched random access, LOWN improves task-normalized quality by 4.9–13.0%; visual routing cuts FLOPs 27–68%, and adaptive acceptance removes another 21–43% of remaining cost.The schedule comparison reports task-normalized quality; the routing and acceptance ablation is on LLaDA-V.Source: §5.3 · p. 9, Table 4 · p. 9, Table 5 · p. 10
  • Unlike pruning, LOWN can later consult the complete image; TEXT-ONLY-unmasked tokens disagree with fresh FULL predictions at 2.2–4.7%, and 92–97% use an anchor from the preceding step.Pruning permanently removes visual information, while LOWN concentrates FULL steps on visually grounded content.Source: §5.3 · pp. 9–10

6. Conclusion

LOwn adaptively schedules image access by checking text-only predictions against saved image-conditioned predictions; across ten benchmarks, it retains 99.6% of LLaDA-V’s score at 0.35× the decoding FLOPs. Reused predictions can differ from fresh FULL predictions, and the method’s cost cannot be set before decoding.

6. Conclusion

  • Across ten benchmarks, LOWN retains 99.6% of LLaDA-V’s score at 0.35× the decoding FLOPs.The same rule transfers to LaViDa and extends to VidLaDA under a controlled video protocol.Source: §6 · p. 10
  • A reused token can differ from a fresh FULL prediction, and this happens more often as the anchor ages.Image dependence also persists more in chart and document reading, where more steps need a FULL forward.Source: §6 · p. 10
  • LOwn’s cost cannot be set before decoding; visual token pruning partly addresses this because its retention ratio provides some cost control.LOwn’s agreement rule determines how many FULL forwards a response needs.Source: §6 · p. 10

A.1. Measurement Protocol

  • The response trajectory is held to the baseline by letting only FULL forwards decide which positions are unmasked.TEXT-ONLY outputs are logged but never enter the decoding state.Source: §A.1 · p. 14

A.2. Per-Benchmark Observations

LLaDA-V shows substantial but benchmark-varying agreement between FULL and TEXT-ONLY predictions, while LaViDa shows lower image dependence on most question-answering benchmarks under the same schedules.

A.2. Per-Benchmark Observations

  • 71.8–92.1% of unmasked decisions agree between FULL and TEXT-ONLY forwards, while 74.0–90.2% of final tokens were already top-1 before unmasking on LLaDA-V.The latter measure is lowest on LLaVA-Bench, CapArena, and DeCapBench.Source: §A.2 · p. 14, Table 6 · p. 14
  • Figure 7 maps, for all ten benchmarks, the share of masked positions where FULL and TEXT-ONLY top-1 predictions disagree across token position (front to back) and denoising step (early to late).Source: Figure 7 · p. 14

A.3. Measurement on LaViDa

  • On LaViDa, image changes an unmasked decision on 1.3–30.6% of steps, versus 7.9–43.8% on LLaDA-V.For LaViDa, FULL and TEXT-ONLY agree on 81.6–98.7% of unmasked decisions, and 78.6–93.7% of final tokens were already top-1 before unmasking.Source: §A.3 · p. 15, Table 7 · p. 15

B. Method Details and Cost Accounting

The appendix specifies LOWN’s decoding procedure, forward-cost accounting, and evaluation settings. It also explains how LOWN’s adaptive costs compare with fixed pruning budgets and how competitor settings are selected.

B.1. Detailed Decoding Algorithm

  • LOwn scans masked positions in confidence order, reuses the anchor for a certified prefix, and refreshes the FULL anchor when certification fails.A repetition guard stops the prefix before a candidate matching an already unmasked neighbor or a token accepted earlier in the scan.Source: §B.1 · p. 15

B. Method Details and Cost Accounting

  • LOwn’s relative cost is f + σρ, combining the frequency of FULL forwards, relative step count, and TEXT-ONLY-to-FULL forward-cost ratio.A FULL step is charged for both forwards; ρ is computed from the analytical FLOPs cost C(n).Source: §B.2 · p. 15, B. Method Details and Cost Accounting · p. 15

B.2. Cost of One Forward

  • The forward-cost model includes quadratic attention, which makes the TEXT-ONLY cost ratio smaller than the token-length ratio when visual tokens are removed.The count excludes the one-time vision encoder and non-matrix-multiplication operations.Source: §B.2 · pp. 15–16

B.3. Other Methods in the Same Form

  • Pruning fixes its budget before decoding, whereas LOWN chooses its FULL-forward frequency and relative step count per input and step.Their combined relative cost is πf + σρ; key-value caching lowers the cost of each forward that reuses the cache.Source: §B.3 · p. 16

C.1. Benchmarks and Settings

  • LLaDA-V evaluations use task-specific response lengths and step counts, and all methods within each reported comparison use the same scorer.The settings include greedy decoding at temperature zero; scorers include task parsers and specified language models.Source: §C.1 · p. 17, §C.2 · p. 17

C.3. Baseline Settings

  • Competitors are compared at the cheapest tested setting whose FLOPs are at least LOWN’s, with scores excluded from setting selection.Figure 1c instead matches scores and uses each pruning method’s cheapest measured setting reaching LOWN’s score, where available.Source: §C.3 · p. 17

C.4. Video Evaluation

The video evaluation uses a controlled VidLaDA protocol on EgoSchema and MLVU, and the timing appendix measures per-example generation time on LLaDA-V.

C.4. Video Evaluation

  • The video evaluation uses 32 frames per sample, a 128-token response, 128 baseline diffusion steps, and block length 32.It evaluates 500 EgoSchema questions and 502 MLVU test questions; VidLaDA’s released evaluation uses 64 frames and two-token decoding instead.Source: §C.4 · p. 17
  • EgoSchema is scored by accuracy and MLVU by mean accuracy over nine task types, using identical questions and frames for both methods.The reported FLOPs ratio excludes the vision encoder.Source: §C.4 · p. 18

D.1. Generation Time

  • LLaDA-V generation time is measured at batch size 1 on NVIDIA H200 GPUs, with CUDA synchronized around the generation call.The timing interval includes decoding, multimodal preparation, and vision encoding; the FLOPs count excludes the vision encoder.Source: §D.1 · p. 18

D.2. Quality and Generation Time

At matched generation time, LOWN keeps the highest average on LLaDA-V; additional experiments examine cache composition, fixed-retention pruning, fewer denoising steps, and LaViDa. The supplied LaViDa passages identify a FLOPs-matched comparison but do not report its outcomes.

D.2. Quality and Generation Time

  • With every competitor given at least LOWN’s generation time, LOWN retains 99.6% and 98.3% of LLaDA-V’s average in the two blocks, versus competitor maxima of 94.6% and 92.8%.It scores highest on seven and eight of ten benchmarks, respectively.Source: §D.2 · p. 18, Table 10 · p. 19

D.3. Cache Configuration and Composition

  • The combined implementation keeps Fast-dLLM’s cache settings, maintains separate image-conditioned and TEXT-ONLY prefix caches, and unions its selected tokens with LOWN’s accepted tokens.Fast-dLLM uses a block spanning the configured output, refreshes the prefix every 32 forwards, and selects tokens with factor 1.Source: §D.3 · p. 19

E.1. Fixed-Retention Pruning Compositions

  • Adding LOWN to the unpruned model or a fixed pruner retains much of its standalone compute reduction, with average quality close to each reference configuration.Table 11 reports relative gains against each corresponding configuration without LOWN.Source: §E.1 · p. 20, Table 11 · p. 20

E.2. Fewer Denoising Steps

  • On CapArena, LLaVA-Bench, and DeCapBench, LOWN uses FLOPs between the T/2 and T/4 settings and scores above both on all three benchmarks.It scores above T/2 even when that setting uses 0.500× FLOPs.Source: §E.2 · p. 20, Table 12 · p. 20

E.3. LaViDa

  • The LaViDa experiments show all ten benchmarks and provide a FLOPs-matched comparison at LOWN’s budget, but these passages do not state the comparison’s results.Table 13 uses each competitor’s cheapest measured setting whose FLOPs are at least LOWN’s.Source: Figure 8 · p. 21, §E.3 · p. 21, Table 13 · p. 21

E.4. Qualitative Samples

Qualitative examples compare LOWN with visual token pruning and show a case where LOWN saves little on a DocVQA table question. The examples highlight that pruning can omit image details needed for an answer, while LOWN’s full forwards cover the complete image.

E.4. Qualitative Samples

  • In the x-axis example, RedVTP prunes the tick 4.0 needed to answer the question and gives fluent but incorrect answer 2.5, while LOWN’s full forwards cover the complete image.The Figure 10 inset enlarges the region around tick 4.0; blue underlines mark tokens unmasked on FULL steps, and other tokens are unmasked on TEXT-ONLY steps.Source: §E.4 · p. 21, Figure 10 · p. 23
  • Across three short-answer and three captioning examples, RedVTP gives incorrect answers or image-inconsistent descriptions, while LOWN matches LLaDA-V’s answers and describes the same main objects.Figure 9 marks correct answers and image-consistent content in green, and incorrect or inconsistent content in red.Source: §E.4 · p. 21, Figure 9 · p. 22

E.5. Limitation Example

  • On the DocVQA table-number example where LOWN saves least, it runs FULL forwards on 14 of 15 steps and costs 0.90× baseline FLOPs, versus 0.45× across the benchmark.LO​​WN answers the question correctly.Source: §E.5 · p. 23, Figure 11 · p. 23

F. Additional Ablations and Analysis

The ablations separate disagreement-first ordering from whole-prefix acceptance, while the certificate analysis measures agreement coverage, prediction error, and anchor age. The reported results show substantial reductions from both acceptance choices and quantify certificate behavior across benchmarks.

F.1. Order of FULL-Step Unmasking

  • Disagreement-first ordering removes 14–35% of visual-routing-only cost while scoring higher on seven of ten benchmarks; whole-prefix acceptance removes a further 2–19%.Table 14 reports scores and relative FLOPs, with each reduction measured from the preceding setting.Source: §F.1 · p. 23, Table 14 · p. 24

F.2. Certificate Reliability

  • At anchor age one, certificates cover 43–72% of eligible positions; 𝜖(∆) rises from age one to four on every benchmark, while acceptance error stays below 𝜖(1).On DocVQA, LOWN costs less than the oracle schedule because adaptive acceptance ends the trajectory early.Source: §F.2 · p. 24, Table 15 · p. 24
  • Coverage is the share of eligible positions accepted by TEXT-ONLY agreement with a one-step-old anchor; acceptance error measures accepted tokens a fresh FULL forward would predict differently.The analysis uses logged baseline trajectories for coverage, 𝜖(∆), and Rref, and LOWN’s own decoding for acceptance error and anchor age.Source: Table 15 · p. 24, §F.2 · p. 24
  • Anchor age counts LOWN steps since the FULL forward that produced an anchor; Table 16 reports accepted-token shares, including a pooled row across benchmarks.Table 15 defines the oracle schedule’s cost relative to the baseline, including one initial FULL forward.Source: Table 16 · p. 24, Table 15 · p. 24
Loading 2610.36671v1…