PaperlayerView exact revision on arXiv

Source-linked AI summary

V-LUMEN: Visual Lookup Memory for Embedding Scaling in Vision-Language Models

Miso Choi, Daekeun Kim, Eunji Kim, Jungbeom Lee

arXiv:2610.37019v1cs.LGcs.AIcs.CLcs.CV

TL;DR

VLMs lack a straightforward lookup-based scaling route because visual features are continuous, spatially structured, and query-dependent. V-LUMEN adds non-parametric visual memory through discretized, spatially and text-conditioned keys; it improves frozen-backbone benchmark performance and matches adapters at matched budgets while scaling capacity in storage.

  • Problem

    Can visual representation capacity grow without increasing per-token compute, despite visual features being continuous, spatially structured, and query-dependent?

    Source: §1 · p. 1

  • Method

    V-LUMEN discretizes visual features with a frozen codebook, forms spatially aggregated and text-conditioned hash keys, and fuses retrieved embeddings into the backbone through gated residual updates.

    Source: §1 · pp. 1–2

  • Results

    V-LUMEN improves a frozen Qwen3-VL-2B backbone on five reasoning-intensive benchmarks, transfers zero-shot to three disjoint-image benchmarks, and matches LoRA or MoE adapters at matched budgets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    V-LUMEN provides an independent capacity axis whose table size can grow without increasing per-token compute, with lookup latency staying constant across a 40× range.

    Source: §1 · p. 2

    The visual-memory-only setting uses a longer training schedule than adapter-based settings, so the paper does not compare it directly with LoRA-only or MoE-only.

    Source: §3.2 · p. 7

Abstract

from arXiv · show

Embedding-level scaling expands the capacity of language models through lookup-based memory rather than by increasing model depth, width, or dense computation. Extending this idea to vision-language models (VLMs) is challenging because visual features are continuous, spatially structured, and query-dependent, unlike discrete text tokens that naturally support n-gram lookup. We introduce V-LUMEN, a visual lookup memory for VLMs. V-LUMEN discretizes visual features with a frozen codebook, builds spatially aware memory keys through spatial aggregation-based hashing, and modulates them with text-conditioned relevance signals; all addressing is non-parametric, and the retrieved embeddings are fused into the backbone through a gated residual pathway. On five reasoning-intensive benchmarks, V-LUMEN substantially improves a frozen Qwen3-VL-2B backbone and transfers zero-shot to three benchmarks whose images are disjoint from the training data. At a matched parameter budget, moving part of a LoRA or MoE adapter's capacity into V-LUMEN preserves accuracy on two backbones, indicating that external visual memory is an independent capacity axis rather than a stronger adaptation method. Because lookup cost is constant in table size and incurred only once at prefill, this capacity grows in storage rather than in per-token compute. Code is available at https://github.com/miso-choi/V-Lumen.

1 Introduction

Dense backbone scaling increases compute at every forward pass, while visual lookup is difficult because VLM features are continuous, spatially structured, and query-dependent. V-LUMEN addresses this with non-parametric visual memory and improves a frozen backbone across reasoning benchmarks while matching adapters at a matched budget.

1 Introduction

  • Visual lookup is challenging because VLM features are continuous, spatially arranged, and question-dependent, unlike discrete text tokens suited to n-gram lookup.Prior lookup memories such as Engram [1], STEM [2], and SCONE [3] address discrete token IDs.Source: §1 · p. 1
  • V-LUMEN discretizes visual features with a frozen codebook, builds spatially aware and text-conditioned hash keys, and fuses retrieved embeddings through gated residual updates.Memory-side addressing is non-parametric; only memory tables and the fusion layer receive gradients.Source: §1 · pp. 1–2
  • V-LUMEN improves frozen Qwen3-VL-2B on five reasoning-intensive benchmarks, transfers zero-shot to three benchmarks with disjoint image sources, and matches smaller LoRA or MoE adapters at matched budgets.The matched-budget comparison also holds on InternVL3.5-2B for the MoE setting.Source: §1 · p. 2
  • Lookup latency stays constant across a 40× range of table sizes, so capacity grows in storage rather than per-token compute.The memory adds a one-off prefill cost without changing per-token decode latency.Source: §1 · p. 2

2 Method

V-LUMEN uses sparse visual-memory lookup to add capacity independently of backbone adaptation. It converts visual features into reusable, spatially and text-conditioned addresses, then gates retrieved embeddings into the backbone; lookup latency remains nearly constant as table size grows.

2.1 Overview

  • V-LUMEN retrieves sparse per-position embeddings and conditionally fuses them into backbone hidden states through a residual pathway.Its three components discretize patches, build region-aware keys, and route the same patch to query-adaptive slots.Source: §2.1 · pp. 2–3, Figure 1 · p. 3

2.2 Visual Embedding Scaling with External Memory

  • A frozen codebook maps continuous patch features to discrete visual codes, creating reusable memory addresses rather than sparse raw-feature keys.The paper adopts the pretrained LaVIT codebook [4] and keeps it frozen.Source: §2.2 · p. 3

2.3 Spatial Aggregation-based Hashing (SAH)

  • Spatial aggregation adds row- and column-level context and coarse positional cues so memory keys represent both local structure and absolute image location.Axial summaries use a mode when a code repeats and a deterministic hash when all codes in the row or column are distinct.Source: §2.3 · p. 4

2.4 Text-Conditioned Hashing (TCH)

  • Text-conditioned hashing modulates spatial keys with query relevance, allowing the same image patch to address different memory slots for different questions.DRH uses text–visual dot-product scores and an EMA-based prior correction before rank-bucketing.Source: §2.4 · p. 5

2.5 Conditional Memory Fusion

  • Each visual token retrieves a fixed number of cells from multi-order, multi-head tables, while total accessible capacity can grow with table size.Retrieved vectors are fused through cell-wise conditional gates weighted by the current hidden state.Source: §2.5 · p. 5

2.6 Retrieval Latency

  • Lookup latency changes little as memory tables scale because key construction does not access the table and gathering takes under 0.5 ms.The latency test varies rows per cell from 0.2× to 8× the default, a 40× range.Source: §2.6 · pp. 5–6
  • Across a 40× table-size range, lookup latency varies by less than 1% (11.16–11.23 ms), and gather time by at most 6% (0.46–0.49 ms).Table 1 reports one fusion layer with 64 visual tokens; most lookup time is spent constructing keys, not accessing the table.Source: §2.6 · p. 6, Table 1 · p. 6

3 Experiments

The experiments train V-LUMEN in two stages and evaluate it on reasoning-intensive benchmarks using a held-out evaluation split. Results show gains with a frozen backbone, matched-budget performance comparable to larger adapters, and zero-shot improvements on benchmarks with image sources absent from training.

3.1 Experimental Settings

  • The training corpus emphasizes recurring visual patterns across documents, tables, OCR regions, spatial structures, diagrams, and natural scenes.Mid-training forms visual static priors, while SFT focuses on reasoning alignment after memory retrieval.Source: §3.1 · p. 6
  • Evaluation uses the held-out 90% of each benchmark after reserving 10% for few-shot training, with relaxed accuracy for numeric and normalized text answers.The reported benchmarks are ChartMuseum, FRIEDA, VisRes, MMMU, and MMMU-Pro; up to 1,000 examples per benchmark are sampled.Source: §3.1 · pp. 6–7
  • The two-stage training updates trainable configuration paths while keeping the backbone frozen, using answer-only next-token cross-entropy.For V-LUMEN combined with LoRA or MoE, adaptation modules and fusion parameters are fine-tuned jointly.Source: §3.1 · p. 7

3.2 Experimental Results

  • +17.79 ± 0.47pp average accuracy gain across five benchmarks when V-LUMEN is attached to the frozen vanilla backbone.No backbone weights are updated, though this setting uses a longer training schedule than adapter-based settings and is not directly compared with them.Source: §3.2 · p. 7
  • At matched budgets, MoE + V-LUMEN reaches +16.92 ± 0.91pp versus +16.85 ± 0.95pp for MoE-only, while LoRA + V-LUMEN reaches +21.15 ± 0.02pp versus +20.58 ± 0.85pp for LoRA-only.The differences are within about one seed standard deviation; the authors do not claim lookup memory improves over adaptation at a fixed budget.Source: §3.2 · p. 7, Table 2 · p. 7
  • V-LUMEN improves over the frozen backbone on all three zero-shot benchmarks, whose image sources are disjoint from training and use no in-domain few-shot samples.The evaluation uses 1,000 samples per benchmark and reports mean ± standard deviation over three seeds.Source: §3.2 · p. 8, Table 3 · p. 8

4 Analysis

The analysis uses a logit lens to track how intermediate-layer predictions compare with each model’s final prediction and the gold answer. V-LUMEN tends to bring predictions closer to the final output earlier, but the pattern is small relative to seed variation.

Method

  • The logit lens projects each layer’s last-token state through the final RMSNorm and tied LM head, measuring JSD to the model’s final prediction and gold-answer top-1 accuracy.No probe parameters are trained; layers 12–27 are shown.Source: §4 · p. 8, Figure 2 · p. 9

Setup

  • The analysis selects VisRes with MoE and MMMU-Pro with LoRA from improving pairs, so these probes are not a random sample of benchmarks.The LoRA comparison uses rank-matched LoRA(0.5) models with and without V-LUMEN.Source: §4 · p. 8, Figure 2 · p. 9

Results

  • Across layers 19–26, JSD to the final prediction is lower with V-LUMEN by −0.036 on VisRes and −0.004 on MMMU-Pro.On VisRes, the gold answer becomes top-1 earlier, with leads of +7.4 and +4.4 points at layers 19 and 20 in two of three seeds.Source: §4 · p. 8, Figure 2 · p. 9
  • At the final layer, top-1 accuracy rises from 0.236 to 0.251 on VisRes and from 0.277 to 0.289 on MMMU-Pro.These match the margins reported in Tab. 2.Source: §4 · p. 8
  • The observed pattern is a tendency, not a consistent effect: only two of three seeds per pair show lower JSD, and mean differences lie within one standard deviation.Both models remain at maximum JSD and zero top-1 accuracy below layer 19.Source: §4 · p. 8

5 Related Work

Prior embedding-scaling methods use lookup over discrete language tokens, while visual features are continuous, spatially structured, and query-dependent. Related routing work motivates non-parametric alternatives to learned routers, which V-LUMEN applies to memory addressing.

5.1 Embedding Scaling in LLMs

  • Language-model lookup methods scale embedding capacity through sparse memory or token and n-gram representations, sometimes storing embeddings off accelerator.These approaches rely on discrete language tokens, so they do not directly transfer to continuous, spatially structured, query-dependent visual features.Source: §5.1 · p. 9

5.2 Hash-based and Deterministic Routing

  • Sparse-MoE learned routers are associated with documented load-imbalance collapse and auxiliary balancing losses.The cited discussion names Switch Transformer as an example.Source: §5.2 · p. 9
  • Hash Layers, BASE Layers, and THOR replace learned expert routing with non-parametric routing schemes.V-LUMEN’s text-conditioned hashing adapts this pattern to memory addressing, routing the same patch to different slots under different queries.Source: §5.2 · p. 10

6 Conclusion

V-LUMEN extends lookup-based scaling to visual features through discrete, spatially aware, and query-conditioned memory addressing. Its results support external visual memory as an independent capacity axis that grows in storage rather than per-token compute.

Conclusion

  • V-LUMEN combines a frozen codebook, non-parametric spatial aggregation, and text-conditioned hashing to address visual memory.The same patch can route to different memory slots depending on the query.Source: §6 · p. 10
  • The memory improves a frozen backbone, transfers zero-shot to unseen image domains, and matches LoRA and MoE adapters at the same total budget.Part of the capacity resides in external memory rather than adapter parameters, and capacity grows in storage rather than per-token compute.Source: §6 · p. 10
  • Sparse hashing-based reads open a path toward serving visual capacity outside GPU memory.Source: §6 · p. 10

7 Future Work

V-LUMEN’s external-memory deployment advantages remain proposed rather than measured, and several limits of the current implementation motivate future work. The section also identifies directions for improving memory use, extending evaluation, and clarifying the method’s relation to prior designs.

7 Future Work

  • Additional memory rows do not yet improve accuracy, while wider retrieved vectors do; training data and key design may limit how much of a larger table is used.Future work would investigate how to make the row-capacity axis deliver gains without increasing per-token reads.Source: §7 · p. 10, A Limitations · p. 15
  • Testing backbones beyond 2B parameters and memory reuse after vision-encoder or connector updates would probe scale and systematic shifts in visual codes.The component ablation motivates evaluating these settings.Source: §7 · p. 10

A Limitations

  • Host- or storage-backed memory serving remains unmeasured; first-layer retrieval cannot overlap with computation, and reported latency reflects GPU-resident tables and implementation overhead.Key construction accounts for about three quarters of prefill overhead in the current implementation.Source: §7 · p. 10, A Limitations · p. 15

B Broader Impact

  • Potential deployment benefits include lower accelerator-memory demands, hardware costs, and energy use, but broader access could also facilitate misuse and sensitive-data retention.These benefits remain potential because such deployments have not been measured; practical use should follow application-specific safety, fairness, privacy, and oversight requirements.Source: B Broader Impact · p. 15

C Extended Related Work; D.5 Distinction from Cross-Modal Query Bridges

  • V-LUMEN uses row/column summaries and coordinate salts for non-parametric memory addressing, unlike related spatial methods that transform features.Its text-conditioned hashing changes external-memory addresses rather than learning a cross-modal connector that transforms or selects visual features.Source: C Extended Related Work · p. 15, §D.5 · p. 18

D.1 Projected Relevance Hashing: Implementation Details; D.2 Debiased Relevance Hashing (DRH) Details

  • PRH rank-buckets projected text-to-patch relevance scores, while DRH debiases patch probabilities with an exponential moving average; their normalized scores can be mixed before bucketing.At inference, the DRH prior is frozen; the default PRH/DRH mixing weight is η=0.5.Source: §D.1 · p. 16, §D.2 · p. 16

D.3 Spatial Key Refiners

  • When a row or column has no repeated code ID, a deterministic rolling hash of sorted IDs supplies its summary, keeping spatial keys defined and order-independent.The rolling hash replaces the undefined mode for all-distinct slices and is intended to preserve content invariance and scale compatibility.Source: §D.3 · pp. 16–17

D.4 Rolling-Hash Index Map for the V-LUMEN Memory Tables

  • The rolling-hash index map uses fixed random coefficients and distinct prime table sizes by order and head; experiments are reported insensitive to this hash choice.The memory tables rely on a fixed, collision-balanced map.Source: §D.4 · p. 17

E Implementation Details

The implementation freezes the backbone and trains selected adaptation and memory components under specified settings. Training schedules vary by configuration, and the V-LUMEN-only setting uses a longer schedule than adapter-based settings.

E Implementation Details

  • V-LUMEN is injected at layers 2 and 8, with a 0.75 learnable residual scaling initialization; PRH/DRH ensembles scale the PRH branch by 0.5.For the main Tab. 2 results, the adaptation budget ratio is k=0.5.Source: E Implementation Details · p. 19

E.1 Compute Resources

  • Experiments used one NVIDIA RTX 6000 Ada GPU; a V-LUMEN-only run took about 5.5 hours, adapter-based runs about 1.5 hours, and all reported experiments required about 700 GPU-hours.Evaluation on the five benchmarks took about two hours per run.Source: §E.1 · p. 19

E.2 Training Schedules

  • V-LUMEN-only training uses a longer schedule than LoRA- and MoE-based settings, so the paper does not compare those settings directly.Within each LoRA or MoE pair, both settings share the same schedule; Table 4 lists training steps by setting.Source: §E.2 · p. 19, Table 4 · p. 19

F Evaluation Protocol

The zero-shot evaluation checks image overlap between the training corpus and three evaluation benchmarks. The authors report disjointness through partition checks or source-level distinctions, depending on the benchmark.

F Evaluation Protocol

  • The authors verify that the three zero-shot benchmarks share no images with the training corpus, addressing image exposure separately from annotation overlap.Their QA pairs are distinct from the training annotations.Source: F Evaluation Protocol · p. 19
  • For TextVQA validation, the authors train on the TextCaps train split and confirm by image-ID check that no validation image appears in training.TextCaps reuses TextVQA images and follows the same train/validation partition, making the sets disjoint by construction.Source: F Evaluation Protocol · p. 19
  • VizWiz-VQA and OCR-VQA are considered image-disjoint at the source level because their collection pipelines do not draw on the cited natural-image or structured-image training sources.The authors state that no image-ID audit is required for these two benchmarks.Source: F Evaluation Protocol · p. 19

G Budget Matching Against Baselines

The section describes budget-matched comparisons with LoRA and MoE, including a rank-matched LoRA design that adds V-LUMEN to the same adapter. In that design, average accuracy is unchanged within seed spread, with benchmark-level differences.

G Budget Matching Against Baselines

  • The rank-matched LoRA design keeps the adapter identical and adds V-LUMEN on top, isolating the memory contribution while using a larger total trainable-parameter budget.The total-budget-matched design controls for that larger combined budget.Source: G Budget Matching Against Baselines · p. 20, Table 6 · p. 20
  • Table 5 reports rank-matched LoRA Design 1 accuracy (%) on Qwen3-VL-2B as mean ± std over three seeds; both settings share LoRA(0.5) (rank 107), with V-LUMEN added on top.Avg ∆ is the average gain over the vanilla backbone in pp.Source: Table 5 · p. 20
  • The total-budget-matched design scales LoRA-only to the combined trainable-parameter count of LoRA(0.5) plus V-LUMEN; the resulting baseline is within 0.2% of the variant’s total budget.This is the design used in Tab. 2.Source: G Budget Matching Against Baselines · p. 20, Table 7 · p. 20
  • For MoE, the baseline matches accessible and per-token activated parameters; the V-LUMEN variant is slightly below it on both budget axes.The comparison uses the same decoder layers and top-1 routing without a shared expert.Source: G Budget Matching Against Baselines · p. 20, Table 8 · p. 21, Table 9 · p. 21

H.1 End-to-End Latency

V-LUMEN adds a one-off prefill cost while leaving per-token decode latency unchanged; at matched memory budgets, it retains performance across two backbones, and its gains persist with another frozen codebook and across tested hyperparameter settings.

H.1 End-to-End Latency

  • V-LUMEN adds 19–25 ms of prefill latency but changes decode latency by at most 0.05 ms against the vanilla, MoE(1.5), and rank-matched LoRA(0.5) baselines.Most prefill overhead comes from CPU-loop key construction; the memory-table read takes about 1.8 ms.Source: §H.1 · pp. 21–22, Table 10 · p. 21, Table 11 · p. 22

H.2 Memory Overhead

  • At matched budgets, adding V-LUMEN increases GPU memory by 0.003 GB for the LoRA pair and 0.02 GB for the MoE pair.Smaller adapters offset nearly all of the memory table’s footprint.Source: §H.2 · p. 22, Table 12 · p. 22

I.1 Choice of Visual Codebook

  • With SEED, V-LUMEN improves over the frozen backbone by +18.51 ± 1.00pp on average, 0.72pp above LaVIT and within the seed spread.SEED uses a codebook half the size of LaVIT’s, and the results indicate gains are not tied to one codebook.Source: §I.1 · p. 23, Table 13 · p. 23

I.2 Hyperparameter Sensitivity

  • Single-axis perturbations yield gains between +17.20 and +18.64pp, within −0.6 to +0.9pp of the default; moving fusion layers to (4, 10) is the only perturbation below it.The default gain is +17.79 ± 0.47pp.Source: §I.2 · p. 23, Table 14 · p. 24

I.3 Generalization to Another Backbone: InternVL3.5-2B

  • On InternVL3.5-2B, V-LUMEN alone improves the frozen backbone by +14.90pp on average, and the budget-matched MoE + V-LUMEN pair is within −0.32pp of MoE-only.The standalone vanilla-based setting uses a longer training schedule and is not directly comparable to adapter-based settings.Source: §I.3 · pp. 23–24, Table 15 · p. 24

I.4 Component Ablation and Robustness to Code Changes

Under a re-drawn visual projection, the full hashing design is more robust than simpler key designs, while tested natural image degradations leave all variants robust. Fusion at early layers also yields stronger gains than moving the second fusion layer deeper.

I.4 Component Ablation and Robustness to Code Changes

  • Without perturbation, variants reach similar average gains of +17.79 to +18.67pp, within the seed spread; rolling hashing alone captures most of the gain.Under JPEG compression and downscaling, all variants change by less than 1pp; the authors do not evaluate encoder or connector updates or small adversarial perturbations.Source: §I.4 · pp. 24–25
  • Under a re-drawn projection, the full model has the smallest accuracy drop at both perturbation levels (−1.61pp and −2.17pp), retaining a +15.62pp gain when every code changes.Rolling hash only loses 4.67pp at 50% and 14.69pp at 100%; the ordering holds at both levels.Source: §I.4 · pp. 24–25, Table 16 · p. 25
  • The proposed explanation for the full model’s robustness is row sharing across codes and contexts, but the authors have not verified this hypothesis directly.The stress test uses a single draw of the perturbed projection.Source: §I.4 · p. 25

I.5 Fusion-Layer Ablation

  • Fusing at layers (2, 6) and (2, 8) yields comparable gains (+17.98 ± 0.28 vs. +17.79 ± 0.47 pp), while moving the second fusion layer deeper generally reduces the gain.The smallest reported gain is +15.08 ± 0.56pp at layers (3, 15); the authors use (2, 8) by default.Source: §I.5 · p. 25, Table 17 · p. 26

J Capacity Scaling: Memory Width and Depth

Increasing memory depth adds storage without changing lookup computation per token, but did not improve accuracy in the tested range; increasing width raised accuracy while also increasing per-token memory reads.

J Capacity Scaling: Memory Width and Depth

  • Across a 15× parameter range from 44.2M to 673.4M, increasing memory depth keeps average gains between +17.79 and +18.36pp, without adding per-token computation.The additional rows did not translate into higher accuracy in this setting.Source: J Capacity Scaling: Memory Width and… · p. 26, Table 18 · p. 26
  • Average gain rises from +16.44pp at dhead = 8 to +18.70pp at dhead = 64, with the +2.3pp difference coming mostly from MMMU (+4.8pp) and MMMU-Pro (+5.2pp).Width also increases memory read per token; dhead = 32 is the chosen balance, as doubling to 64 adds 0.9pp at twice the memory parameters.Source: J Capacity Scaling: Memory Width and… · p. 26, Table 18 · p. 26

K Logit-Lens Analysis Details

This section presents a limited logit-lens analysis of V-LUMEN against adapter-only models and notes the absence of human-subject research or important, original, or non-standard LLM components in the core method.

K Logit-Lens Analysis Details

  • The probes cover four adapter-benchmark pairs where V-LUMEN’s mean exceeds the adapter-only mean; open-ended FRIEDA and ChartMuseum are analyzed with JSD only.The pairs are MMMU-Pro (LoRA), VisRes (MoE), FRIEDA (LoRA), and ChartMuseum (MoE); the first two use JSD and gold top-1 accuracy.Source: K Logit-Lens Analysis Details · p. 27
  • −0.006±0.011 in both FRIEDA and ChartMuseum: V-LUMEN slightly lowers mean JSD over layers 19–26, with differences small relative to seed variance.FRIEDA also shows lower JSD around layers 15–17 and 22–23 (−0.01); ChartMuseum briefly rises at layer 16 (+0.012 ± 0.001) before falling below baseline around layers 23–25.Source: Figure 3 · p. 27, K Logit-Lens Analysis Details · p. 27
  • The analysis uses the main-evaluation samples, and text-conditioned hashing is applied only to single-image samples, excluding 7% of MMMU-Pro and 61% of FRIEDA samples.Sample counts are 900 each for MMMU-Pro and VisRes, 888 for ChartMuseum, and 438 for FRIEDA; V-LUMEN’s final-layer top-1 prediction matches the main evaluation’s first generated token for every sample.Source: K Logit-Lens Analysis Details · p. 27

2. Limitations

  • The paper acknowledges that offloading the memory is a design property that it does not measure.Source: §2 · p. 29
  • The paper marks IRB approval as not applicable because the research does not involve human subjects.Source: §2 · p. 34
  • The core method uses no LLM for training-data generation or annotation, and evaluation relies on rule-based accuracy rather than LLM-based judging.The paper describes Qwen3-VL-2B-Instruct and InternVL3.5-2B as standard pretrained backbones augmented by V-LUMEN.Source: §2 · p. 35

Related papers

Loading 2610.37019v1…