PaperlayerView exact revision on arXiv

Source-linked AI summary

MemLoRA: Distilling Expert Adapters for On-Device Memory Systems

Massimo Bini, Ondrej Bohdal, Umberto Michieli, Zeynep Akata, Mete Ozay, Taha Ceritli

arXiv:2512.04763v1cs.LGcs.CLcs.CV

TL;DR

Memory-augmented systems depend on large models for memory operations, while small models are less capable and caption-based visual handling limits direct image reasoning. MemLoRA uses task-specific adapters on small models, with a vision extension and VQA benchmark augmentation. Its text and visual evaluations report strong results, including gains from dedicated visual adapters.

  • Problem

    Memory systems rely on costly large models, small models lack sufficient performance, and caption-based image handling does not evaluate direct visual reasoning.

    Source: §1 · pp. 1–2

  • Method

    MemLoRA uses separately specialized adapters on small models for memory operations, and MemLoRA-V adds a vision expert and LoCoMo VQA tasks.

    Source: §1 · pp. 2–3

  • Results

    MemLoRA surpasses Gemma2-27B on three of four student-teacher combinations, while dedicated adapters raise InternVL3-2B's V score from 70.8 to 81.3.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    The reported approach supports local memory operations with small models and extends memory systems to native visual understanding.

    Source: §1 · p. 2, §5 · p. 8

Abstract

from arXiv · show

Memory-augmented Large Language Models (LLMs) have demonstrated remarkable consistency during prolonged dialogues by storing relevant memories and incorporating them as context. Such memory-based personalization is also key in on-device settings that allow users to keep their conversations and data private. However, memory-augmented systems typically rely on LLMs that are too costly for local on-device deployment. Even though Small Language Models (SLMs) are more suitable for on-device inference than LLMs, they cannot achieve sufficient performance. Additionally, these LLM-based systems lack native visual capabilities, limiting their applicability in multimodal contexts. In this paper, we introduce (i) MemLoRA, a novel memory system that enables local deployment by equipping SLMs with specialized memory adapters, and (ii) its vision extension MemLoRA-V, which integrates small Vision-Language Models (SVLMs) to memory systems, enabling native visual understanding. Following knowledge distillation principles, each adapter is trained separately for specific memory operations$\unicode{x2013}$knowledge extraction, memory update, and memory-augmented generation. Equipped with memory adapters, small models enable accurate on-device memory operations without cloud dependency. On text-only operations, MemLoRA outperforms 10$\times$ larger baseline models (e.g., Gemma2-27B) and achieves performance comparable to 60$\times$ larger models (e.g., GPT-OSS-120B) on the LoCoMo benchmark. To evaluate visual understanding operations instead, we extend LoCoMo with challenging Visual Question Answering tasks that require direct visual reasoning. On this, our VLM-integrated MemLoRA-V shows massive improvements over caption-based approaches (81.3 vs. 23.7 accuracy) while keeping strong performance in text-based tasks, demonstrating the efficacy of our method in multimodal contexts.

1. Introduction

Memory systems address the need to retain user-specific information across conversations, but reliance on large cloud models limits private, offline, and resource-constrained deployment. MemLoRA and MemLoRA-V target these constraints with specialized adapters and direct visual capabilities, alongside a LoCoMo VQA augmentation.

1. Introduction

  • Memory-augmented dialogue systems need to retain user-specific knowledge across sessions, which ordinary context windows do not provide.The limitation matters when users expect systems to remember prior interactions, preferences, and contextual details.Source: §1 · p. 1
  • The overview routes extraction, memory update, and memory-augmented generation through distinct adapters, with text-only or multimodal generation selected by input.The figure frames this specialization as enabling local execution without cloud API dependence.Source: Figure 1 · p. 2
  • Large-model dependence requires cloud calls, adding latency and cost while limiting on-device use in privacy-sensitive, offline, or resource-constrained settings.MemLoRA replaces large-model API queries with a small on-device model equipped with task-specific adapters.Source: §1 · p. 2
  • Caption-based image handling can lose fine-grained details, spatial relationships, and numerical information, while original LoCoMo evaluation does not test direct visual reasoning.Its image questions rely on captions rather than image access during inference.Source: §1 · p. 2
  • MemLoRA equips small models with specialized adapters for memory operations, while MemLoRA-V adds native visual capabilities and a specialized vision adapter.The paper also augments LoCoMo with VQA tasks requiring direct image access.Source: §1 · pp. 2–3

2. Related Work

Prior memory systems use retrieval, persistent memory structures, and LLM-driven operations, while distillation and parameter-efficient finetuning offer routes to smaller, adaptable models. MemLoRA combines small models with expert adapters to run memory operations locally.

Memory-Augmented LLMs

  • Many memory approaches use agentic frameworks to orchestrate extraction, updating, and retrieval through iterative LLM queries; these require multiple computationally expensive queries and do not prioritize on-device deployment scenarios.Source: §2 · p. 3

Knowledge Distillation with LLMs

  • Output-only distillation uses generated text rather than teacher logits, enabling black-box teachers and allowing outputs to be cleaned or reformatted before training.The paper adopts this approach for its practical advantages.Source: §2 · p. 3

Parameter Efficient Finetuning Methods

  • LoRA and related PEFT modules can be swapped across tasks while retaining one base model, supporting efficient adaptation and avoiding inference-time latency overhead.MemLoRA applies this modularity to expert memory adapters on small language and vision-language models.Source: §2 · p. 3

3. Method

MemLoRA replaces the large model in a persistent-memory pipeline with a small model using specialized adapters, then extends the approach to native visual processing and direct visual-question evaluation. The method builds on Mem0’s extraction, update, and memory-augmented generation stages.

3.1. Preliminaries

  • Mem0 extracts useful knowledge from dialogue, updates stored memories, and provides retrieved memories as context for personalized generation.Its repeated large-model calls make the original approach impractical for on-device deployment.Source: §3.1 · pp. 3–4

3.2. Our Method: MemLoRA

  • MemLoRA combines Mem0 and LoRA, replacing the large model with a smaller deployable model and assigning separate adapters to extraction, update, and generation.The adapters are trained from teacher-generated text outputs rather than logits, and the small model switches between experts at inference.Source: §3.1 · p. 4, §3.2 · p. 4
  • Training data is tailored by operation: extraction outputs are cleaned, update examples are filtered to new extractions, and generation uses ground-truth LoCoMo responses.The generation expert uses ground truth rather than teacher outputs because teacher accuracy is described as around 40–50%.Source: §3.2 · p. 4
  • Experts are trained independently with next-token prediction and integrated into a deployment pipeline that dynamically loads the adapter for each memory operation.Extraction, update, and generation retain Mem0’s three-stage flow while enabling fully local execution.Source: §3.2 · pp. 4–5, Figure 2 · p. 5

3.3. Native Visual Understanding Capabilities

  • MemLoRA-V replaces the base model with a small vision-language model and adds a VQA expert; image queries activate that expert instead of the language generation adapter.The extension addresses captioning’s loss of uncaptured image details and lack of native on-the-fly image querying.Source: §3.3 · p. 5
  • The LoCoMo augmentation tests counting, color identification, and unusual-object questions chosen for high validator error rates among eight candidate types.Single-word answers support word-similarity evaluation, while answer and reason fields both train the vision expert.Source: §3.3 · pp. 5–6, Figure 3 · p. 6
  • The multimodal contribution comprises Mem0-V, on-device MemLoRA-V with a vision expert, and a LoCoMo VQA augmentation for evaluating visual reasoning.The benchmark is designed to assess direct image reasoning rather than caption-only responses.Source: §3.3 · pp. 5–6

4. Experiments

Experiments evaluate MemLoRA and MemLoRA-V on LoCoMo text QA and a new visual-question task, comparing memory operations and efficiency against Mem0 baselines. Adapter-equipped small models achieve strong benchmark results, with efficiency and ablation analyses examining the trade-offs and contributions of specialization.

4.1. Experimental Setup

  • Evaluation uses LoCoMo’s 10 multi-session dialogues and three memory operations; results use a held-out 20% test split without hyperparameter tuning.The QA task includes single-hop, multi-hop, temporal, and open-domain questions; the split is approximately 70/10/20% for training, validation, and testing.Source: §4.1 · p. 6

4.2. Benchmark Results

  • On LoCoMo, MemLoRA reaches J 47.2 versus Gemma2-27B’s 39.1; MemLoRA-V reaches V 81.3 versus caption-based Mem0’s 23.7.For InternVL3-2B, the V score rises from 70.8 to 81.3 with a dedicated adapter; the caption-based baseline’s highest V score is 23.7.Source: §4.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7

4.3. Efficiency Measures

  • Under the same resources, MemLoRA uses 10–20× less memory and delivers 10–20× faster responses than LLM-powered Mem0 at comparable performance.Efficiency measures are averaged over knowledge extraction, memory update, and memory-augmented generation on a portion of LoCoMo.Source: Table 3 · p. 7, §4.3 · pp. 7–8

4.4. Ablations

  • Extraction and update experts trained on filtered teacher-generated data outperform the teacher, while QA-task specialization gives generation its largest gain: J 47.2 versus 39.1.The generation result is reported for the Gemma2-2B expert against its Gemma2-27B teacher, with high-quality ground-truth data noted as a condition for exceeding the teacher.Source: Table 4 · p. 8, §4.4 · p. 8
  • Increasing student size initially improves MemLoRA performance, but the gains diminish as students grow larger.The scale ablation reports the largest improvements for the smallest models.Source: §4.4 · p. 8, Table 5 · p. 8

5. Conclusions

MemLoRA uses specialized adapters to run memory operations on small models locally, and MemLoRA-V extends the system with native visual understanding. The authors report strong text and visual benchmark performance alongside efficient, privacy-preserving deployment.

5. Conclusions

  • MemLoRA equips small models with specialized memory adapters for efficient on-device execution of memory-augmented systems.The conclusion presents local execution as an alternative to large cloud-based systems.Source: §5 · p. 8
  • On LoCoMo, the adapters achieve performance comparable to models 10–60× larger, while ablations show experts consistently surpass teacher models.The authors also report diminishing performance returns as student model size increases.Source: §5 · p. 8
  • MemLoRA-V adds native visual understanding through a specialized vision expert adapter and extends LoCoMo with challenging VQA tasks.The augmented benchmark is presented as a new benchmark for multimodal memory-augmented systems.Source: §5 · p. 8
  • The authors conclude that lightweight specialized memory systems can support privacy-preserving, efficient deployment on mobile and edge platforms.Source: §5 · p. 8

Supplementary Material

The supplementary material describes how MemLoRA prepares teacher data and reduces prompt overhead for small models. Specialized adapters learn memory operations from examples, allowing shorter input prompts while preserving performance.

A. Teacher Data Preparation and Efficiency

A.1. Input Instruction Prompts Reduction

  • MemLoRA replaces long extraction instructions with a minimal conversation-context prompt, relying on the specialized adapter to learn the task from training examples.The approach is motivated by the limited context windows and reasoning capacity of small models.Source: §A.1 · p. 12
  • MemLoRA reduces average input tokens 7.4× for knowledge extraction (756.94 to 102.94 tokens) and 10× for memory update (1734.62 to 172.56 tokens).Table A6 compares average input-token use for the two operations between Mem0 and MemLoRA.Source: §A.1 · p. 14, Table A6 · p. 14

A.2. Output Data Cleaning

MemLoRA fine-tuning is described as producing cleaner, more focused outputs than Mem0, reducing unnecessary response text. Reported output-token comparisons show model-dependent verbosity for Mem0 and controlled lengths for MemLoRA.

A.2. Output Data Cleaning

  • MemLoRA fine-tuning produces cleaner, more focused outputs without unnecessary verbosity, reducing parsing overhead and generation token use.The passage attributes the cleaner responses to the fine-tuning approach.Source: §A.2 · p. 15
  • In the illustrated memory-update example, the cleaned response keeps the two ADD entries rather than repeating unchanged old memories.Source: §A.2 · p. 16
  • Gemma2-2B generates 54.75 average output tokens for MemLoRA memory updates, while GPT-OSS-120B with Mem0 generates 289.25 tokens for knowledge extraction.Table A7 compares average output-token consumption across models and systems.Source: Table A7 · p. 17, §A.2 · p. 17

B. Training Pipeline

The training pipeline prepares teacher-generated supervision, searches for specialized expert adapters, and then selects and tests an integrated pipeline. Validation targets end-to-end benchmark performance, with final evaluation on held-out conversations.

B. Training Pipeline

  • The three-phase pipeline prepares teacher-student data, searches expert-adapter hyperparameters with nested validation, then integrates and tests optimized adapters.The pipeline is designed to maximize end-to-end task performance on LoCoMo.Source: B. Training Pipeline · p. 18
  • LoCoMo is split by conversation into about 70% training, 10% validation, and 20% test data, with teacher outputs generated for training and validation.The split is described as preventing leakage between splits.Source: B. Training Pipeline · p. 18
  • Adapter search uses teacher-forcing validation loss for early stopping, then full-pipeline validation on benchmark metrics to select candidates for end-to-end performance.For extraction and update, the validation pipeline uses the base student model for future untrained stages.Source: B. Training Pipeline · p. 18
  • Top candidate adapters are assembled across stages, alternative combinations are validated, and the selected combination is evaluated on the held-out test set.The final test set consists of conversations reserved from hyperparameter tuning.Source: B. Training Pipeline · p. 18

C. Creating the VQA benchmark

The benchmark adds challenging, objectively answerable one-word visual questions to LoCoMo images. InternVL3-2B identifies challenging question categories, and InternVL3-78B generates questions for the dataset splits.

C.1. Selection of Question Instructions

  • Questions are designed to be challenging, answerable with one word, and have one objectively correct response.The instructions illustrate the format with a question about which hand holds a glass.Source: §C.1 · p. 21
  • InternVL3-2B identifies categories 2, 3, and 4 as particularly challenging; InternVL3-78B generates questions in those categories across LoCoMo’s splits.Images appearing in multiple splits are removed, and the augmented splits are used for training, hyperparameter selection, and evaluation, respectively.Source: §C.1 · p. 21

C.2. Qualitative Visualization of Visual Questions and Answers

  • Figure C4 shows LoCoMo images with generated questions, InternVL3-78B answers, and InternVL3-2B predictions with and without expert adapters.The figure caption identifies the compared predictions but does not state their relative performance.Source: Figure C4 · p. 22

D.1. Implementation Details

The implementation follows Mem0’s modular memory architecture and uses locally executed models and tools. Experiments compare large teacher models with small student models, including small vision-language models for visual tasks.

D.1. Implementation Details

  • The memory system follows Mem0’s modular design for extraction, updating, and retrieval, using mxbai-embed-large embeddings and FAISS with Euclidean distance.Models are downloaded locally and run at temperature 0.0 for deterministic outputs.Source: §D.1 · p. 23
  • Pipeline stages are evaluated separately with independent memory banks to prevent information leakage; experiments use single 80GB NVIDIA A100 GPUs except for InternVL3-78B.InternVL3-78B requires distributed inference across 3–4 GPUs with tensor parallelism.Source: §D.1 · p. 23

D.2. Model Architectures and Configurations

  • Gemma2-2B and Qwen2.5-1.5B serve as student language models, while GPT-OSS-120B and Gemma2-27B serve as teacher models.GPT-OSS-120B is also used as the LLM-as-a-judge evaluator.Source: §D.2 · p. 23
  • InternVL3-78B supplies teacher-generated targets for VQA, while InternVL3-1B and InternVL3-2B are the student vision-language models.The student variants support both language-only and vision-language modes.Source: §D.2 · p. 23

D.3. Expert LoRA Adapters Training

The paper uses LoRA adapters with different layer placements for memory extraction and updating versus generation. Adapter training uses shared LoRA initialization and core settings, with stage-specific training durations in practice.

D.3. Expert LoRA Adapters Training

  • Extraction and update adapters target query and value projections in every attention layer, while generation adapters target all linear layers.The reported preliminary evaluations found these placements sufficient for extraction and update and better for generation, respectively.Source: §D.3 · p. 23
  • All LoRA adapters use rank 8 and alpha 16, with random A and zero B initialization so training begins at pretrained-baseline performance..Source: §D.3 · p. 23
  • Training allows up to 50 epochs with early stopping, uses BF16, and typically runs longer for update adapters and least long for generation adapters.The setup fixes dropout at 0.1 and uses AdamW with weight decay 0.01.Source: §D.3 · p. 23

D.4. Evaluation Configuration

Evaluation combines text-similarity metrics with an LLM-based factual-accuracy judgment. The judge labels answers against gold responses, while VQA accuracy uses exact string matching after normalization.

D.4. Evaluation Configuration

  • The composite score L combines ROUGE-1, METEOR, BERTScore-F1, and SentenceBERT, while judge score J is the primary factual-accuracy metric.L is computationally efficient; J requires substantially greater computational resources.Source: §D.4 · p. 24
  • GPT-OSS-120B judges whether generated answers are CORRECT or WRONG against gold answers, returning a JSON label after a one-sentence rationale.The evaluation prompt is run at temperature 0.0 for reproducibility.Source: §D.4 · p. 24
  • Judge grading accepts answers that address the same topic as the gold response and treats equivalent date formats or references to the same time period as correct.The prompt says to grade generously when a longer generated answer touches on the gold answer’s topic.Source: §D.4 · p. 24
  • VQA accuracy is exact string match after lowercasing and removing leading and trailing whitespace.Source: §D.4 · p. 24

E.1. VLM performance at different scales

MemLoRA-V adapters let InternVL3-2B achieve strong LoCoMo and visual-question-answering results, while larger VLMs perform progressively better but do not fit on device. Comparisons also show that specialized adapters shift the language-only versus multimodal performance pattern at larger model scales.

E.1. VLM performance at different scales

  • Larger InternVL3 models (8B and above) show progressively stronger performance, but do not fit on device; InternVL3-78B requires at least 3 GPUs.Table E8 evaluates LoCoMo using L and J and the visual task using VQA accuracy.Source: §E.1 · p. 25, Table E8 · p. 25
  • InternVL3-2B+Exp achieves the highest L (44.6) and J (40.3) scores among tested configurations up to 38B parameters, with 81.3% VQA accuracy, higher than the 8B model.The 78B model reaches 92.0% VQA accuracy but requires distributed inference across at least 3 GPUs.Source: §E.1 · p. 25, Table E8 · p. 25

E.2. Abilities as memory systems for LLM and VLM

  • With Mem0, language-only models outperform multimodal counterparts on text-based LoCoMo J and L scores, while MemLoRA’s specialized adapters reverse the J-score comparison for 1.5B models.InternVL3-2B scores 40.3 on J versus 36.6 for Qwen2.5-1.5B; at 0.5B language-component scale, the language-only advantage remains.Source: §E.2 · p. 25

Related papers

Loading 2512.04763v1…