PaperlayerView exact revision on arXiv

Source-linked AI summary

One Token per Multimodal Evidence: Latent Memory for Resource-Constrained QA

Zhi Zheng, Ziqiao Meng, Hao Luan, Wei Liu, Wee Sun Lee

arXiv:2606.10572v1cs.AI

TL;DR

External-memory QA is costly because raw text and images consume substantial generation tokens and storage. Latent Memory compresses each evidence item into one retrievable latent token, achieving competitive text-QA performance with 3× fewer tokens and reducing multimodal generator cost by up to 10×.

  • Problem

    Raw text and image evidence imposes substantial token, storage, latency, and context-window costs for resource-constrained external-memory QA.

  • Method

    Latent Memory compresses each text or image evidence item into one latent token in a unified space for retrieval and frozen-generator answer generation.

  • Results

    Latent Memory achieves competitive text-QA performance with 3× fewer tokens and the strongest average-F1 trade-off on image-grounded WebQA while reducing generator cost by up to 10×.

  • Takeaways & Limitations

    Latent Memory provides an efficient alternative to token-level memory for scenarios requiring fast responses and low storage pressure.

  • Takeaways & Limitations

    The approach assumes evidence can be decomposed into independently compressed and retrieved text or image units, limiting preservation of global structural dependencies.

Abstract

from arXiv · show

External memory effectively grounds large language models (LLMs) and vision-language models (VLMs)-based question answering (QA) in relevant multimodal evidence. However, existing memory paradigms represent each memory item in raw text and image forms, so retrieval-based systems must pass the retrieved text or images to the generation LLMs/VLMs, resulting in high token consumption and storage pressure, making it unaffordable for resource-constrained applications. We propose Latent Memory, a latent-space memory paradigm that replaces each raw text or image evidence item with a single high-dimensional latent token produced by a small compressor LLM/VLM. Rather than retrieving raw evidence for generation, Latent Memory operates in a unified latent representation space: the query is embedded into this space to retrieve relevant latent tokens, and the retrieved latent tokens are directly prompted to a pretrained LLM or VLM for answer generation. To make each latent token simultaneously informative for reconstruction, retrieval, and generation, we train the compressor with reconstruction, contrastive, and distillation objectives in a unified end-to-end manner. Latent Memory is evaluated on seven text-only QA benchmarks (e.g., HotpotQA) and multimodal QA benchmarks, where it achieves competitive QA performance compared to advanced RAG baselines while consuming 3x to 10x fewer generator tokens. It can also deliver the strongest image-grounded QA performance on WebQA. Code is available at https://github.com/zz1358m/Latent-Memory-Master.

1 Introduction

Latent Memory addresses the token, latency, and storage costs of prompting generators with uncompressed multimodal evidence by replacing each evidence item with one retrievable latent token. It jointly trains these tokens for reconstruction, retrieval, and generation, achieving competitive QA performance with improved efficiency across text-only and multimodal benchmarks.

  • Core framework: Latent Memory compresses each multimodal evidence item into a single high-dimensional latent token for resource-constrained QA.The token can replace raw evidence during generation while reducing token consumption and storage pressure.
  • Core framework: The unified latent representation and retrieval space enables latent tokens to support both contextual representation and retrieval.Queries are embedded into the same space to retrieve relevant latent tokens.
  • Training objective: A combined reconstruction, contrastive, and distillation objective trains tokens to preserve evidence information, align supporting queries, and support answer generation.A small LLM/VLM compressor is fine-tuned so latent tokens retain raw information, retrieval capability, and generation behavior.
  • Empirical scope: Latent Memory improves the efficiency-performance trade-off across eight text-only and multimodal contextual QA benchmarks and four generator LLM/VLMs.The framework achieves competitive generation performance while using compressed latent evidence instead of raw text or images.

2 Related Work

Prior work grounds QA with external memory and RAG, compresses lengthy text evidence into continuous tokens, and explores latent reasoning and unified multimodal representations. Latent Memory extends these directions by encoding, retrieving, and prompting multimodal evidence in one latent representation space.

  • External Memory and RAG: External-memory QA uses full-context prompting or RAG, with retrieval methods encoding queries and evidence into a shared space for nearest-neighbor search.Multimodal embedding methods additionally retrieve images using embedding models.
  • Evidence Compression: Evidence-compression methods reduce lengthy text memories into fewer continuous tokens, while xRAG and CLaRa also use latent representations for retrieval.These methods focus on text-based situations.
  • Latent-Space Reasoning: Latent reasoning research shows that models can communicate through continuous latent states, while related work seeks unified embeddings for multimodal comprehension and generation.Latent Memory applies this principle to encode, retrieve, and prompt multimodal evidence in a unified latent representation.

3 Methodology: Latent Memory

Latent Memory replaces raw multimodal evidence with one latent token per item, enabling retrieval and generation through a compact continuous interface. A compressor is trained with reconstruction, contrastive retrieval, and generation-distillation objectives so each token supports these roles simultaneously.

  • Memory Compression: Each evidence item is compressed into a single latent token, and the original raw evidence is discarded from the resulting memory.A small compressor LLM/VLM produces the token from a learnable [MEM] embedding and its final hidden state.
  • Retrieving and Generating: Latent tokens are projected into a retrieval space, selected for a query, and mapped into the frozen generator’s hidden space as continuous evidence tokens.This changes the interface from prompting retrieved raw evidence to directly prompting retrieved latent tokens.
  • Reconstruction: Optional reconstruction recovers text, image captions, or CLIP image embeddings from latent tokens without reconstructing raw image pixels.Recovered CLIP embeddings can condition the diffusion-based image generator unCLIP for rough image recovery.
  • Compressor Training: Three complementary objectives train one token to preserve evidence content, support evidence selection, and match generator behavior.Reconstruction recovers text or image-related representations, contrastive learning separates supporting from irrelevant evidence, and distillation aligns latent-conditioned generation with raw-context generation.
  • Contrastive Retrieval Loss: Contrastive training pulls query representations toward supporting latent memories and pushes them away from sampled negative evidence in a shared multimodal retrieval space.Because text and image evidence use the same retrieval space, the objective supports unified retrieval over mixed multimodal memory.

4 Experiments

Latent Memory is evaluated for text-only and multimodal contextual QA against context-based and retrieval-augmented baselines under frozen LLaMA, Mistral, and LLaVA generators. Across these settings, it aims to preserve competitive QA and retrieval performance while substantially reducing generator-side token usage, including on image-grounded WebQA.

  • Experimental settings: Experiments cover text-only contextual QA on HotpotQA, 2WikiMultihopQA, and MuSiQue, plus multimodal contextual QA on WebQA.The same HotpotQA-trained memory is also tested without additional tuning on the two out-of-domain text datasets.
  • Baselines: Baselines span full-context and evidence-compression methods alongside sparse, dense, embedding-based, and multimodal retrieval systems.Reported methods include LLMLingua, xRAG, CLaRa, BM25, Dense Retrieval, Qwen-3-Embedding, Nemo Retriever, and Qwen-3-VL-Embedding.
  • Text-only QA: At k=5, the 1B Latent Memory uses 71 generator tokens, versus 208 for Dense Retrieval, while achieving competitive EM/F1 and the strongest reported out-of-domain Recall@k.These results are reported under the fixed 8B LLaMA generator setting.
  • Text-only QA: At k=5, Ours-1B uses 74 generator tokens and reaches the strongest out-of-domain Recall@k with competitive F1 under a frozen Mistral-7B-Instruct generator.This setting directly compares against pretrained Mistral-based latent-context baselines such as xRAG and CLaRa.
  • Multimodal QA: At k=5, one-token Latent Memory is strongest on WebQA’s image-grounded subset while using far fewer generator tokens than raw-evidence retrieval baselines.WebQA evaluation separately reports image-grounded n=2,511 and text-grounded n=2,455 subsets under a frozen LLaVA-1.5-13B generator.
  • Multimodal QA: Latent Memory’s unified representation yields Recall@K that is 10%+ higher than Dense Retrieval on text-grounded and especially image-grounded questions.Raw-image prompting can exceed the generator’s pretrained context window, producing poor-quality outputs such as meaningless content or blanks.

5 Discussion

The discussion shows that reconstruction is important to Latent Memory’s retrieval and generation quality, while additional latent-token budget improves performance without sacrificing token efficiency. A case study illustrates interpretable multimodal grounding, and the authors identify limitations when evidence meaning depends on global structure rather than atomic units.

  • Ablations and Analysis: Removing reconstruction, especially negative evidence, reduces retrieval accuracy and answer quality, with larger EM/F1 declines than Recall@k.The ablation supports using negative evidence to anchor the unified latent representation space.
  • Better Latent Memory Capability with more Token Budget: 8-token Latent Memory surpasses Qwen-3-Emb-0.6B on out-of-domain EM/F1 at each k while using fewer generator tokens.The out-of-domain average covers 2WikiMultihopQA and MuSiQue.
  • A Representative Case Study: In an image-grounded WebQA example, retrieved latent evidence supports the correct answer and preserves the counting information required by the question.Optional reconstruction indicates that latent tokens retain interpretable semantic content for both text and image evidence.
  • Limitation on Current Modality Coverage and Future Directions: The current design independently compresses and retrieves atomic text or image units, limiting coverage when meaning depends on global structure.The authors identify complex tables as requiring row-column relations and layout information.

6 Conclusion … B.3 Hyperparameter Settings

Latent Memory compiles evidence into latent tokens for unified retrieval and generation, achieving competitive QA performance with substantially lower generator-token consumption. The paper also details its related work, prompt and projection pipelines, training choices, and future extensions to additional modalities.

  • 6 Conclusion: Latent Memory compiles each evidence item into one latent token, retrieves latent memories through a compressed query, and feeds them directly to a frozen generator.The same latent representation supports retrieval and generation across text-only and multimodal QA settings.
  • 6 Conclusion: 3× fewer tokens: text QA achieves competitive EM/F1 performance compared to RAG baselines while reducing generator consumption.The conclusion reports this reduction for text QA relative to RAG baselines.
  • 6 Conclusion: Latent Memory targets fast-response, low-storage scenarios and is planned for extension to complex tables and complex videos.The paper focuses on external evidence and does not consider Agentic Memory.
  • A Related Work: Related work spans sparse and dense RAG, multimodal embedding retrieval, discrete and latent context compression, retrieval-compression interaction, and latent-space modeling for LLMs.The appendix organizes these topics across the related-work and implementation contents.
  • A.1 RAG and Embedding Retrieval: Raw-evidence RAG methods retrieve text or images but still pass native evidence to the generator, so accurate retrieval does not remove per-evidence token cost.This includes BM25, dense text retrieval, VisRAG, and multimodal embedding methods.
  • A.2 Evidence Compression for Generation: Compression methods reduce generation context, whereas xRAG and CLaRa combine retrieval with compression but remain limited by their representation or text-only scope.xRAG uses a two-stage bridge, while CLaRa requires substantial pre-training and is text-only.
  • A.3 Retrieval-Compression Interaction: Latent Memory extends unified retrieval and generation from text to multimodal scenarios, improving generation-token efficiency and reducing storage pressure.The paper contrasts this capability with xRAG and CLaRa and situates it within broader latent-space representation work.
  • B Implementation Details: The implementation uses matched teacher-student prompts, normalized 512-dimensional retrieval vectors, projected latent inputs to frozen generators, and reconstruction, distillation, and retrieval-time training choices.Teacher prompts use raw evidence, student prompts use one latent token per retrieved item, and inference retrieves top-k vectors with FAISS before inserting them through inputs_embeds.

B.4 Time Complexity Analysis

Latent Memory reduces generator-prefill cost by replacing raw evidence with T latent tokens, making complexity depend on retrieved evidence rather than the full corpus. Without precompilation, it adds a parallelizable evidence-compilation cost while retaining lower online input length than raw-evidence RAG.

  • Comparison with Full Context: With precompilation, Latent Memory uses O((|Q| + kT)2dϕ) generator-prefill complexity, independent of N.Each retrieved evidence item contributes T latent tokens rather than ¯L raw tokens.
  • Comparison with Raw-Evidence RAG: Latent Memory sends |Q| + kT tokens after retrieval, versus RAG’s |Q| + k¯L, yielding the smallest online generator-prefill complexity when T ≪¯L and usually k ≪N.RAG embeds all evidence items but still forwards k raw evidence items to the generator.
  • Comparison with Full Context: Full Context incurs O((|Q| + N ¯L)2dϕ) generator-prefill complexity, which grows quadratically with N when ¯L is fixed.The full evidence pool is sent to the frozen generator.
  • Deployment Settings: Without precompilation, Latent Memory adds O(N ¯L2dθ) evidence compilation, which is linear in N and highly parallelizable.The compressor compiles the evidence pool into latent memories before generation.

B.5 Storage Complexity Analysis · C Additional Experiments and Discussion

The storage analysis shows that Latent Memory’s main text-side benefit is lower generator token usage, while image-side latents reduce persistent storage and online activation memory. Additional experiments examine token counts, domains, compressors, generators, and training objectives, finding that 8-token multimodal memory surpasses the best RAG baseline out of domain.

  • B.5 Storage Complexity Analysis: On the language side, short raw snippets can be smaller than high-dimensional latent vectors, so Latent Memory primarily reduces generator-side token usage rather than persistent corpus storage.The comparison counts only stored evidence representations, excluding small metadata fields.
  • B.5 Storage Complexity Analysis: On the image side, replacing each extracted WebQA image with one LLaVA latent token provides a clear storage reduction and avoids storing raw visual evidence.The official extracted WebQA images occupy about 75GB in total, while latent image storage uses 4096 bf16 values per item.
  • B.5 Storage Complexity Analysis: A 4096-dimensional bf16 latent token is more storage efficient than an uncompressed RGB image above roughly 53 × 53 pixels.With a separately stored 512-dimensional fp32 retrieval key, the threshold shifts to about 59 × 59 pixels.
  • B.5 Storage Complexity Analysis: Latent Memory reduces online activation memory from O(k¯Ldϕ) for raw multimodal RAG to O(kTdϕ) for projected latent tokens when ¯L ≫T.This pattern applies because raw multimodal RAG instantiates visual embeddings for each retrieved image, whereas Latent Memory instantiates only projected latent tokens.
  • C Additional Experiments and Discussion: One-token Latent Memory can expose a token-performance trade-off in multimodal settings while replacing the raw retrieved context.The experiments vary latent evidence representation in multimodal QA.
  • C Additional Experiments and Discussion: 8-token Latent Memory provides a better trade-off and surpasses the best RAG baseline on out-of-domain performance.This result is reported for replacing raw evidence with 8-token Latent Memory.
  • C Additional Experiments and Discussion: The appendix studies how many latent tokens are needed and whether performance generalizes across evidence domains, compressors, generators, and training objectives.RQ-A1 varies tokens per evidence item; RQ-A2 tests broader benchmarks, compressors, and generator settings; RQ-A3 ablates reconstruction and negative reconstruction objectives.

C.1 Token Count Ablation · C.2 Generalization Ability on More Domains

The token-count ablation shows that more latent tokens improve answer quality while preserving efficiency, despite limited Recall@k gains. Cross-domain tests find favorable text-QA trade-offs and competitive multimodal generalization under frozen generators.

  • C.1 Token Count Ablation: The ablation varies each evidence item from 1 to 8 latent tokens while keeping evidence units and retrieval granularity unchanged.The default configuration uses one token, with 2-, 4-, and 8-token variants.
  • C.1 Token Count Ablation: Retrieval still uses one key per evidence item by pooling its multiple latent tokens before the retrieval projection.Thus, the ablation isolates the effect of additional latent capacity on generation and token cost.
  • C.1 Token Count Ablation: Increasing the latent-token budget consistently improves EM and F1, with larger-token variants surpassing the main retrieval baselines in answer quality.The reported text-QA analysis covers HotpotQA, 2WikiMultihopQA, and MuSiQue.
  • C.1 Token Count Ablation: Larger latent-token budgets do not produce a comparable improvement in Recall@k at fixed k.Recall trends are summarized for the out-of-domain average over 2WikiMultihopQA and MuSiQue.
  • C.2 Generalization Ability on More Domains: The 8-token Latent Memory method achieves similar performance to BM25 while requiring 2.5 times fewer tokens.On the four additional text-QA benchmarks, Qwen-3-Embedding-based RAG performs best, while 1-token and 8-token variants offer good trade-offs.
  • C.2 Generalization Ability on More Domains: On multimodal SlideQA, Latent Memory improves retrieval coverage and achieves competitive EM with far fewer generator tokens, although Nemo remains stronger on EM/F1.The evaluation uses a frozen LLaVA-1.5-13B generator on a 20-image dataset.

C.3 Core Training Ablation · C.4 Stronger Text Compressors · C.5 Direct Transfer to Similar Generator

Core training ablations show that evidence reconstruction supports both retrieval and generation, while stronger compressors improve answer quality under matched budgets. Latent tokens also transfer directly to a similar LLaMA generator without retraining, with larger budgets improving performance.

  • C.3 Core Training Ablation: Reconstructing sampled negative evidence helps the memory representation, whereas query reconstruction perturbs the intended query-encoder role.The default reconstructs positive and sampled negative evidence, while query reconstruction is disabled.
  • C.3 Core Training Ablation: Removing evidence reconstruction consistently weakens both EM/F1 and Recall@k, indicating that reconstruction stabilizes retrieval and generation together.The retrieval and generation settings remain fixed across the ablation variants.
  • C.4 Stronger Text Compressors: LLaMA-3B is consistently the strongest compressor, improving EM/F1 over LLaMA-1B and Qwen-1.5B at all reported k values.The frozen generator, token budget, and training recipe are held fixed, isolating the compression backbone.
  • C.4 Stronger Text Compressors: At k=5 on HotpotQA, LLaMA-3B improves LLaMA-1B from 34.8/48.9 EM/F1 to 35.4/49.7 while maintaining similar high Recall@k.This dataset-level result demonstrates improved answer quality without a reported retrieval collapse.
  • C.4 Stronger Text Compressors: At k=5 on 2WikiMultihopQA, LLaMA-3B reaches 25.1 EM and 37.3 F1 versus 24.3/36.7 for LLaMA-1B, improving Recall@k from 74.2 to 77.6.The passage describes 2WikiMultihopQA as showing a more visible improvement than HotpotQA.
  • C.5 Direct Transfer to Similar Generator: Latent tokens trained with Meta-Llama-3-8B-Instruct can be reused directly with frozen LLaMA-3.1-8B-Instruct without compressor retraining or latent-token adaptation.The compressor, memory bank, retrieval procedure, and projection interface remain fixed.
  • C.5 Direct Transfer to Similar Generator: At k=5, the 8-token transferred variant reaches 30.7/41.8 average EM/F1, exceeding the strongest raw-evidence embedding baseline while using fewer generator tokens.The transferred setup preserves the one-token efficiency pattern, while larger latent-token budgets provide a clear capacity gain.

C.6 Multimodal Results with Gemma … D.2 Text-only Case Studies

Across multimodal evaluation, retrieval-only ablations, and case studies, Latent Memory combines token efficiency with competitive evidence grounding. The analyses show that latent capacity affects reconstruction, latent representations contribute beyond retrieval quality, and compact retrieval can recover multi-hop answers.

  • C.6 Multimodal Results with Gemma: With frozen Gemma-3-12B-Instruct, Latent Memory achieves the best WebQA-Image Accuracy and second-best F1 while using 10× fewer tokens.The retrieval pool and evidence candidates remain fixed, with Gemma-3-4B-PT as compressor and LLaMA-3.2-1B-Instruct as reconstruction decoder.
  • C.7 Latent Tokens as Retrievers: Latent tokens serve simultaneously as retrieval keys and generator inputs, whereas hybrid as-RAG variants retrieve with latents but generate from raw text or images.The comparison isolates gains from latent-key retrieval versus latent-token generation quality.
  • C.7 Latent Tokens as Retrievers: In text-only QA, Latent Memory with k=1 outperforms Latent-Memory-as-RAG with k=5 at the same token count, showing compression contributes beyond retrieval.The 8-token variant provides a better compression trade-off.
  • C.7 Latent Tokens as Retrievers: In multimodal WebQA, raw-evidence as-RAG performs better on text, but latent-token generation performs better on images, linking image gains to efficient representation.The result indicates that representation efficiency, not retrieval alone, contributes to image-side improvements.
  • D Case Study: The case studies diagnose latent representations qualitatively through reconstruction, text-only multi-hop retrieval, and unified text-image retrieval rather than adding leaderboard evidence.These analyses address three questions about preserved evidence, complete multi-hop chains, and concrete multimodal behavior.
  • D.1 Reconstruction Quality of Latent Tokens: More latent tokens consistently reduce reconstruction cross-entropy, while 1-token representations can achieve low CE but fail to fully reconstruct evidence.Additional latent capacity preserves more evidence information and improves representation faithfulness and interpretability.
  • D.2 Text-only Case Studies: Two text-only multi-hop cases rank the required evidence chains at the top and recover exact-match answers from compact retrieved sets.The cases connect a screenwriter to a Nicolas Cage film and a person to a company’s headquarters through supporting facts.

D.3 More Multimodal QA Case Studies

Additional WebQA case studies demonstrate Latent Memory’s multimodal reasoning behavior, including counting and comparison, using unified text-image retrieval and latent-token conditioning of a frozen LLaVA-1.5-13B generator.

  • Qualitative WebQA behavior: Figure 11 presents additional multimodal WebQA examples covering counting and comparison reasoning.The cases include both successful and challenging examples to illustrate qualitative behavior beyond aggregate results.
  • Retrieval and generation pipeline: Latent Memory retrieves evidence from a unified text-image candidate pool before generation.This retrieval setup is used consistently across the additional case studies.
  • Retrieval and generation pipeline: The frozen LLaVA-1.5-13B generator is conditioned directly on the retrieved latent tokens.Figure 11 uses the same retrieval-and-generation pipeline as the main experiments.

E Baselines, Datasets, and Licenses

The paper evaluates text-only and multimodal QA across a broad dataset suite using controlled generator, retrieval, and baseline settings. It also defines generator-side token accounting and reports upstream license provenance without inferring unstated licenses.

  • Datasets: Text-only experiments train on HotpotQA and evaluate on HotpotQA validation, 2WikiMultihopQA, and MuSiQue without task-specific fine-tuning.The same checkpoint is additionally evaluated on NQ, TriviaQA, Qasper, and WICE for generalization.
  • Datasets: WebQA supports multimodal training and evaluation with separate image-grounded and text-grounded reporting over a unified text-image candidate pool.SlideQA provides a pure visual benchmark covering slides and detail capture.
  • Baseline settings: Text-only baselines use a fixed Meta-Llama-3-8B-Instruct generator and shared question prompts, comparing Full Context, LLMLingua, and BM25 Retrieval.xRAG and CLaRa are compared using their pretrained models in the Mistral-7B-Instruct setting, with CLaRa evaluated at reported 16× compression.
  • Baseline settings: Multimodal baselines retrieve from the same unified WebQA candidate pool and use frozen LLaVA-1.5-13B or Gemma-3-12B-Instruct generator families.The LLaVA family is used in the main WebQA setting, while Gemma is evaluated in Appendix C.6.
  • Evaluation accounting: Generator-side token counts measure the final prompt budget, with k matched across retrieval baselines and corresponding Latent Memory settings.Counts include retrieved text tokens, visually processed multimodal tokens, or projected Latent Memory tokens inserted into the frozen generator.
  • Licenses: License reporting records terms verified from official upstream release pages, marks unstated licenses as not stated, and requires compliance with original content-provider rights for redistributed web content.The summary covers datasets, pretrained models, and baselines.
Loading 2606.10572v1…