PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieved but Not Delivered: Multimodal Memory Delivery for Long-Term Agents

Yuhang Jiang, Qingwei Liao, Kaize Yin, Xingling Liu, Luca Cuomo, Silvio Bacci

arXiv:2609.32590v1cs.CVcs.AIcs.CLcs.IR

TL;DR

Multimodal memory work has not isolated what retrieved evidence reaches the model, or in what form. This paper measures that delivery stage and introduces DeliverMem’s delivery decisions and retrieval-side adapter. Delivery improves MemLens accuracy more than perfect retrieval in the controlled comparison, and DeliverMem leads on both benchmarks.

  • Problem

    Multimodal memory evaluations do not isolate what of retrieved memories reaches the model and in what form.

    Source: §1 · p. 2

  • Method

    DeliverMem preserves original modalities, adds readable identities and seen-times, and uses a retrieval-side adapter for the property delivery cannot supply.

    Source: §1 · p. 3

  • Results

    With gold messages fixed, delivering pixels improves MemLens accuracy by 13.87 points, versus +2.31 points from perfect retrieval on the same questions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DeliverMem leads the strongest published memory agent at all four MemLens context lengths and beats DMV-Bench’s strongest method at every setting on both backbones.

    Source: §1 · p. 3

    The measured modality is image, and delivery granularity such as crop versus whole frame is not evaluated.

    Source: §8 · p. 9

Abstract

from arXiv · show

Work on memory for multimodal agents optimizes what is written, updated and retrieved. Between retrieval and the answer, however, is a stage that multimodal memory evaluations do not isolate: what of the retrieved memory reaches the model, and in what form. We call it delivery, and a controlled decomposition on MemLens locates the remaining room there. With the retrieved evidence set exactly fixed, delivering the original pixels instead of withholding them raises accuracy by 13.87 points on an 8B backbone, whereas making retrieval perfect on those same messages improves it by 2.31. Delivery is the larger term on all three MemLens backbones and grows with backbone strength; retrieval grows too, without closing the gap. We propose DeliverMem, an instantiation of delivery as three decisions: keep the original modality, give each item a readable identity, and state when it was seen, with a retrieval-side adapter for the one property delivery cannot supply. Each is measured against a delivery-matched control that alters only its own variable. DeliverMem leads the strongest published memory agent on MemLens at all four context lengths, and beats DMV-Bench's own strongest method at every setting on both backbones. On MemLens it does this on a tenth to a seventieth of the input. Each decision helps only where the question lacks what it supplies, and is null elsewhere. A single fixed configuration nonetheless leads both benchmarks, without training any component or modifying the stored records. Project page: https://avalon-s.github.io/DeliverMem/

1 INTRODUCTION

Multimodal memory can retrieve the right message yet still fail when its pixels are withheld. This paper isolates delivery—the form in which retrieved evidence reaches the model—and finds more accuracy headroom there than in retrieval.

1 INTRODUCTION

  • The salmon example answers incorrectly with the right message but correctly when its pixels are delivered.The photograph contains the smaller piece’s position, which the user’s text does not mention.Source: §1 · p. 1
  • With gold messages fixed, delivering pixels improves MemLens accuracy by 13.87 points, versus +2.31 points from perfect retrieval on the same questions.Delivery is the larger term on all three tested backbones and grows with backbone strength.Source: §1 · p. 2
  • Delivery determines which information in retrieved memories is usable by the answering model, including modality, identity, temporal position, binding, order, and quantity.The paper measures modality, identity, and temporal position, fixes order in controls, and varies quantity separately.Source: §1 · p. 2
  • DeliverMem keeps original modalities, adds readable identities and seen-times, and uses a retrieval-side adapter for a property delivery cannot supply.It requires no training and does not modify stored records.Source: §1 · p. 3
  • DeliverMem leads published memory agents on MemLens at all four context lengths and beats DMV-Bench’s strongest method at every setting on two backbones.On MemLens, its 32K score matches a frontier proprietary LVLM reading the full conversation while using a tenth of the input.Source: §1 · p. 3

2 RELATED WORK

Prior memory work chiefly addresses what is stored, updated, and retrieved; related multimodal systems also alter storage or retrieval representations. This paper distinguishes delivery by holding the retrieved set fixed and varying what reaches the model.

2 RELATED WORK

  • Memory research has focused on persistence, revision or forgetting, and retrieval, including graph stores and late-interaction matching.The cited survey organizes mechanisms by the stage on which they act.Source: §2 · p. 3
  • Captioning at ingestion discards pixels, while whole-screen visual memory can reduce state-level failures yet worsen action-level ones.Prior work also reports that vision-language models may miss swaps between similar but semantically different images.Source: §2 · p. 3
  • OCR-Memory, PMMC, MemOCR, and SMMBench change retrieval representations, stored content, or benchmarked stages rather than isolating delivery with a fixed retrieved set.MemOCR renders structured text as a compressed image; this paper instead varies what reaches the model while holding stored and retrieved sets fixed.Source: §2 · p. 3
  • Existing agent-memory systems generally act at write or retrieval time, while text-only post-retrieval work has no modality decision.The paper places the nearest multimodal systems on the retrieval side rather than delivery.Source: §2 · p. 3

3 BENCHMARKS

The evaluation uses two public benchmarks that test complementary forms of multimodal memory: recalling one decisive image and answering questions from evidence across long conversations.

3 BENCHMARKS

  • DMV-Bench tests image-level visual recall by asking an agent to identify the product photograph containing an object absent from page text.Scoring is exact, with no judge model; the evaluation uses Qwen2.5-VL-7B and Gemini 2.5 Flash.Source: §3 · p. 4
  • MemLens pairs long multimodal conversations with questions supported by a few scattered messages across four context lengths.The canonical evaluation uses 195 questions, an 8B Qwen3-VL backbone, and the benchmark’s 235B judge.Source: §3 · p. 4
  • The benchmarks are complementary: DMV-Bench asks for a single decisive image, while MemLens asks for facts spread across several messages.The paper compares memory-augmented agents and reports long-context models alongside them because their inference costs differ by more than an order of magnitude.Source: §3 · p. 4

4 SEPARATING DELIVERY FROM RETRIEVAL

A controlled MemLens decomposition finds substantially more headroom in delivery than retrieval, with the delivery advantage persisting across larger backbones. Further controls and coverage analyses qualify where the gains arise.

4 SEPARATING DELIVERY FROM RETRIEVAL

  • The decomposition varies one argument at a time: ΔD removes pixels while retaining gold messages, and ΔR replaces gold messages with retrieved ones while retaining pixels.Both contrasts are paired per question with the answering model, prompt template, and judge held shared across conditions.Source: §4 · p. 4
  • Among 173 non-refusal questions, the retriever includes a gold message for 96.5% but recovers only 20.2% of needed gold messages, versus 99.4% under the oracle.Five times the evidence coverage yields only +2.31; adding an image retrieval channel or removing distractors each improves accuracy by 4.05 points, within noise.Source: §4 · p. 4
  • On MemLens 32K, delivery gains +13.87 while perfect retrieval gains +2.31; using the authors’ retriever, delivery gains +14.45 and the interaction is −0.58, undetectable.Table 1 compares retrieval conditions by row and whether pixels are retained by column.Source: §4 · p. 4, Table 1 · p. 5
  • With identical image slots, a real but wrong photograph gains +0.00 over no photograph, while correct pixels gain +40.98 on image-specific questions and at most +2.86 on the other three types.The right evidence can also be present while the model answers wrongly: on DMV-Bench, 98% of failures selected a neighbour despite the correct item being in context.Source: §4 · pp. 4–5
  • Delivery remains the larger term as backbone size grows: +13.87, +15.61, and +17.92, while retrieval rises from +2.31 to +9.83 and +11.56.The ratio narrows from 6.00× to 1.59× and 1.55×.Source: §4 · p. 5

5 METHOD

DeliverMem combines delivery-side controls on the outgoing context with a retrieval-side adapter for evidence retrieval cannot provide. Its delivery components preserve media, make items addressable, and supply temporal position without changing stored records.

5 METHOD

  • (D1)–(D3) retrofit stacks that retain original media and timestamps by changing only the delivered context; (R1) changes which items arrive.The DMV-Bench adaptation leaves its retrieval, encoding, and store untouched.Source: §5 · p. 5
  • (D1) delivers retrieved items with their original pixels rather than a textual proxy.For MemLens, each item receives the finest temporal index available: a date; DMV-Bench uses append ordinal because it has no timestamps.Source: §5 · pp. 5–6
  • (D2) makes images addressable with readable tags rendered into pixels, and the prompt refers to each memory by its tag.The tag is added after embedding, leaving retrieval ranking untouched; placing it in text instead of pixels costs 7.89 points on DMV-Bench.Source: §5 · pp. 5–6
  • (D3) annotates each delivered item with its position in the memory store, using the finest temporal index that store provides.MemLens uses dates, while DMV-Bench uses an append ordinal rendered as [seen #k of N].Source: §5 · p. 6
  • (R1) retrieves images as a whole plus a 2 × 2 overlapping grid, scoring regions with MaxSim over SigLIP 2 features.It targets evidence confined to a small image region.Source: §5 · p. 6

6 EXPERIMENTS

DeliverMem leads published memory agents across MemLens context lengths and beats DMV-Bench’s strongest published method across settings on both backbones. Its comparisons include benchmark-published pools and a separate paired reproduction.

6 EXPERIMENTS

  • At 32K, MemAgent-7B scores 32.82 while an 8B model with ten delivered messages reaches 51.28, above its 50.77 whole-conversation score.The 51.28 score is level with GPT-5.4 reading the full conversation; at 64K and 128K, listed frontier systems score higher.Source: §6 · p. 6, Table 2 · p. 7
  • At 32K, delivered input is 2,405 tokens versus 50.77 for the 8B model reading the whole conversation?Source: §6 · p. 6
  • DeliverMem leads the published memory systems at every setting on both benchmarks, while DMV-Bench margins against a same-batch DualMem reproduction grow with J on both backbones.The paired margins rise from +7.63 to +12.86 on Qwen and from +12.58 to +22.62 on Gemini; these are distinct from the benchmark-published comparisons.Source: §6 · p. 7, Table 3 · p. 7

7 ATTRIBUTION AND TRANSFER UNDER DELIVERY-MATCHED CONTROLS

Delivery-matched controls isolate the contribution of individual delivery decisions, while a 2 × 2 analysis shows interaction with retrieval. Effects vary by benchmark and question type: each component helps where its information is needed and can be null elsewhere.

7 ATTRIBUTION AND TRANSFER UNDER DELIVERY-MATCHED CONTROLS

  • On DMV-Bench, deployed tags gain 11.85 points over a constant symbol and 9.88 over no tag; on MemLens, the tag contrast is half a point.The tag helps DMV-Bench questions that ask which retrieved frame contains a cue, whereas MemLens questions more often need a date or fact than a name.Source: §7 · p. 8
  • On DMV-Bench, accurate dates improve temporal-question accuracy by 8.76 points over no date and 14.43 over permuted dates; incorrect dates therefore harm performance relative to accurate dates.Accurate dates gain +13.92 on 194 temporal-reasoning questions against value-preserving permutation, while the other 505 questions move by −0.40.Source: §7 · pp. 8–9
  • The image-message binding control loses 6.36 points when only the image-to-message pairing is permuted, at identical gold coverage and image count.Source: §7 · p. 9
  • On DMV-Bench, the 2 × 2 gives (R1) a +8.83 main effect and delivery a +3.96 main effect, with a significant +5.14 interaction.The delivery condition moves the tag and ordinal together, and delivery and retrieval are reported jointly rather than as independent additive contributions.Source: Table 5 · p. 8, §7 · p. 9
  • (R1) is positive on DMV-Bench but null on MemLens, where replacing the default index with the region index moves the same condition by −1.03.The region index targets small image regions; the median cue occupies 1.7% of a DMV-Bench frame. (D3) is positive on MemLens temporal questions and null on DMV-Bench.Source: §7 · p. 9
  • One fixed configuration leads on both benchmarks, with delivery doing the work on MemLens and (R1) doing most of it on DMV-Bench.The components are compared with controls that preserve delivered items and order while changing only the target variable.Source: §7 · pp. 7, 9

8 LIMITATIONS

The study does not measure delivery granularity: whether to send a crop, the whole frame, or both.

8 LIMITATIONS

  • Delivery granularity—crop, whole frame, or both—is unmeasured; separating it from region retrieval requires a benchmark with region annotations.MemLens photographs lack a well-defined relevant region, while DMV-Bench already uses the region axis for retrieval.Source: §8 · p. 9

9 CONCLUSION

The paper identifies delivery as a distinct stage between retrieval and answering, with more accuracy headroom than retrieval on MemLens. DeliverMem makes delivery decisions explicit and leads both benchmarks in one configuration without training.

9 CONCLUSION

  • Delivery has more accuracy headroom than retrieval on MemLens, significantly on Qwen3-VL-8B and in the same direction on two larger backbones.The salmon question was answered wrongly with the right message in the prompt, then correctly once its pixels arrived.Source: §9 · p. 9
  • With the retrieval adapter, one untrained configuration leads both benchmarks; its three delivery decisions are a first pass, with binding and audio/video decisions still outside the interface.Binding is measured at +6.36.Source: §9 · p. 10
  • The authors used generative AI for writing and experimental work, but not to generate, augment, or modify experimental data.They reviewed the work and recomputed their measured accuracy figures from stored per-question scores and per-probe outcomes.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The study uses public benchmarks as released and collects no new data or human-subject data. Delivering stored pixels has a privacy consequence when media are personal, while the interface leaves stored records unchanged.

ETHICS STATEMENT

  • The work involves no human subjects or new data collection; both benchmarks are public and used as released.DMV-Bench’s cued frames are synthetic edits of studio photographs rather than pictures of people.Source: ETHICS STATEMENT · p. 10
  • Delivering stored pixels places more raw imagery before the answering model than a captioning system does, so personal media require a separate retention and re-display decision.D2 tags the delivered copy without altering the stored record.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The appendix documents benchmark and reference-system construction, controlled delivery conditions, and reproducibility details. It situates delivery against prior work and describes how the modality decomposition extends beyond images.

REPRODUCIBILITY STATEMENT

  • The paper reports local and hosted model execution, image and text encoders, and a single 32 GB GPU; end-to-end reruns require benchmark data and harnesses not redistributed.Per-question MemLens scores, DMV-Bench probe outcomes, and traces for worked failures are slated for release; reported table contrasts can be recomputed from those outputs.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 BENCHMARK SETUP

  • DMV-Bench tests whether an agent can use a visual cue in a product image to navigate back to its product, scoring exact product-page matches.Its reference methods are NoMemory, TextOnly, and Caption; probes are nested, and the authors note that treating them as independent inflates apparent significance.Source: §A.1 · p. 15

A.2 WHERE DELIVERMEM ACTS IN EACH BENCHMARK

  • DeliverMem changes only the context-assembly stage on DMV-Bench, while its MemLens comparisons use the paper’s own retrieval baseline.Controls isolate the image set, pixels, tags, dates, item text, order, or image-to-message binding; the shuffled-tag control preserves rank cues from item order.Source: §A.2 · p. 15, Table 6 · p. 15, §C.1 · p. 18

A.3 CHOOSING THE REFERENCE SYSTEM

  • DualMem is strongest across all eight DMV-Bench cells, while MemAgent-7B is the strongest published MemLens agent overall at 32K, scoring 32.82.On MemLens refusal questions, Mem0 scores 77.27 and MemAgent-7B scores 13.64; direct inference with the paper’s backbone reaches 50.77 while reading the full context.Source: §A.3 · pp. 15–16, §A.4 · p. 16

A.5 DIRECT-LVLM POOL ON MEMLENS

  • Direct LVLMs read the whole conversation at more than ten times the inference cost, and DMV-Bench evaluates this family only up to 128K.Table 8 compares direct LVLMs on MemLens with the benchmark’s reported results and the paper’s own row.Source: §A.5 · p. 16, Table 8 · p. 17

A.6 COMPARISON WITH PUBLISHED NUMBERS

  • +7.02 points at J=5 over DMV-Bench’s published number; the sampled J=50 row estimates the exhaustive-probe quantity using about a tenth of its probes.The table reports probe-pooled task success rate and compares DeliverMem with DualMem, the strongest system in the published pool.Source: §A.6 · p. 16, Table 9 · p. 18

B.1 THE POST-RETRIEVAL STAGE IN PRIOR WORK

  • Prior work names post-retrieval reliability and prompt assembly, but the cited prompt-assembly characterization prices cost without measuring its accuracy effect.The paper uses delivery-matched conditions because presentation controls can alter measured effects; cited text-only work and retrieval-focused multimodal systems address different stages.Source: §B.1 · p. 16

B.2 WHERE PIXELS SURVIVE

  • Whether pixels survive depends on the memory unit: discrete-frame systems tend to retain frames, whereas segment- or turn-based systems summarize them at write time.The paper frames its results around images but says the decomposition applies where one modality is rendered into another; audio additionally requires temporal position, aligned timing, and query-specific span selection.Source: §B.2 · p. 17, §C.2 · p. 18

C.3 WHICH CHANNEL CARRIES THE IDENTITY

Controlled delivery comparisons vary the information and form reaching the model while holding the delivered set fixed. The results span image content, item identity, configuration selection, and how much perfect retrieval adds.

C.3 WHICH CHANNEL CARRIES THE IDENTITY

  • Stamping the identity wins by 7.89 points over naming it in text; using both channels together gains +10.97 over neither, with a +6.56 interaction.The pattern recurs at J=5, where delivering identity gains +13.88 against delivering none and the interaction remains positive at +4.81.Source: §C.3 · p. 19
  • A readable identity requires agreement between the tag in the pixels and the instruction in text, rather than either channel alone.The channel comparison keeps the items, order, image set, ordinals, and tag-numbering rule identical.Source: §C.3 · p. 19

C.4 THE DELIVERED CONTEXT

  • The delivered context specifies each condition as bytes; prompt controls keep rank order available while removing only the address.The recalled-item prompt can also state when an item was seen, with a smaller ordinal indicating a longer time ago.Source: §C.4 · p. 20, §D.2 · p. 22

C.5 CONFIGURATION SELECTION

  • Selection raises the four MemLens scores to 54.87, 46.67, 44.10, and 40.51, gains of +3.59, +2.56, +2.05, and +1.54 over the single configuration.Only the 32K gain is significant under a paired test (p = 0.0156).Source: §C.5 · p. 20

D.1 ONE QUESTION END TO END

  • Across five conditions for one salmon question, four answer correctly, while withholding pixels from the padded gold messages yields a score of 0.The gold messages and question remain unchanged in the comparison with pixels withheld.Source: §D.1 · p. 20

D.3 CONTENT OR PRESENCE: COUNTERFACTUAL IMAGES

  • +11.56 is the reported content-specific value of delivering original pixels rather than phase-randomized images on MemLens.Real but incorrect photographs score the same as no photograph; the phase-randomized condition preserves the most properties of the original while destroying objects.Source: §D.3 · p. 23

D.4 WHERE THE EFFECT CONCENTRATES

  • +40.98 is the delivery effect on questions asking for a fact from a particular image, versus +2.86, +0.00, and +0.00 on the other three question types.The effect is concentrated in the 35% of questions a picture can answer; the remaining 65% ask something a picture cannot.Source: §D.4 · p. 23

D.5 WHY PERFECT RETRIEVAL CHANGES SO LITTLE

  • Perfect retrieval adds only +2.31 points overall, while delivery gains +40.98 on image-fact questions and retrieval differences are not significant in the reported split.The retriever already finds the relevant message for image-fact questions; additional gold messages go unused, while retrieval gains fall on questions combining evidence across messages.Source: §D.5 · pp. 24–25

D.6 FAILURE ANALYSIS ON DMV-BENCH

The DMV-Bench failure audit finds that retrieved items are often present in context but the model selects the wrong one. Across MemLens backbones, delivery remains the larger measured term, while the J=50 sampled and exhaustive estimates are close.

D.6 FAILURE ANALYSIS ON DMV-BENCH

  • On Qwen2.5-VL-7B, failures are selection errors rather than retrieval errors; on Gemini 2.5 Flash, 92% of baseline failures are retrieval misses.In the Qwen audit, retrieval recall is 98.4% for the baseline and 100% for DeliverMem, and the correct item is often in context when a neighbour is chosen.Source: §D.6 · p. 25

E.2 SAME-BATCH PAIRED COMPARISON ON DMV-BENCH

  • The same-batch DMV-Bench comparison against DualMem uses seed-averaged estimates and an exact cluster permutation test.The caption gives test floors of 0.002 at k=10 chains and 0.031 at k=6.Source: Table 16 · p. 26
  • At J=50, the Monte-Carlo and exhaustive constructions give +12.79 and +12.86, respectively; the main table reports the sampled row for protocol parity.The Monte-Carlo construction scores ten probes per recall reach per chain, while the exhaustive construction scores every probe.Source: §E.2 · p. 26

E.3 THE SAME 2 × 2 ON GEMINI 2.5 FLASH

On Gemini, retrieval accounts for nearly all of the measured J=50 gain, while delivery contributes little; later controls show that results vary with question type, image granularity, and query conditioning.

E.3 THE SAME 2 × 2 ON GEMINI 2.5 FLASH

  • At J=50 on Gemini, (R1) has a +22.16 main effect, versus +0.46 for delivery; their interaction is +0.31 (p = 0.656).The delivery effect is not significant at p = 0.156, while all six chains agree in sign for (R1) at p = 0.031.Source: §E.3 · p. 26, Table 17 · p. 27

E.4 MARGIN GROWTH WITH MEMORY LOAD

  • As memory load grows, the opponent's score falls by more than three times as much as ours; at 32K, the judge score is 51.28 for both K=10 and K=20.Increasing K from 3 to 10 gains +11.28, while increasing it from 10 to 20 changes the score by zero.Source: §E.4 · p. 27, §E.5 · p. 27

E.6 INPUT COST

  • Delivered size stays near-constant while text falls 24% and visual input rises 127%, as image-bearing messages increase from 3 to 7.The reported input ratio excludes offline indexing work. Most contrasts are paired within a run; at J=5, the variance residual adds 3 to 5.5 points and shrinks slowly as probes increase.Source: §E.6 · pp. 27–28, §F.1 · p. 30, §F.2 · p. 31

E.7 TEMPORAL POSITION ACROSS CONTEXT LENGTHS

  • Across context lengths, the temporal-question contrast gains +10.42, +8.33, +8.33, and +10.42, while gains outside temporal questions are at most +1.36.Accurate dates improve over no date by 8.76 points and over permuted dates by 14.43 on temporal questions; permuted dates score 5.67 below no date.Source: §E.7 · p. 28

E.8 PER-TYPE EFFECTS OF THE ASSEMBLED CONFIGURATION

  • Across 789 released questions, the assembled configuration costs −8.62 points on knowledge update while gaining +7.22 on temporal reasoning.In the final batch, adding the tag and date gains +14.58 on temporal reasoning and loses −13.79 on knowledge update; the overall change is +1.03 points.Source: §E.8 · p. 28

E.9 WHAT (R1) DEPENDS ON

  • On 1,000 cued frames, (R1) raises Recall@1 from 22.9 to 55.6; region MaxSim selects a tile containing the isolated object 99.2% of the time.The isolated object occupies a median 1.7% of the image in 847 frames, and whole-image pooling averages it into the surrounding scene representation.Source: §E.9 · pp. 28–29, Figure 5 · p. 29

E.10 BINDING CONTROL BY QUESTION TYPE

  • Permuting image-message bindings significantly reduces pooled performance across answerable types; removing query conditioning costs 27.69 to 29.23 points against recency selection and 27.18 to 33.85 against random selection.The equal-budget control delivers ten messages through the same pipeline and rendering.Source: §E.10 · p. 29, Table 19 · p. 30, §E.11 · p. 30

F.3 WORKED FAILURE CASES

  • In three audited constant-tag failures, the target was retrieved and correctly named in the trace, but the agent opened a different image; distinct tags gain 11.85 points over the constant symbol.Across audited failures whose trace names an index, the action follows that attribution in 95 to 98% of cases.Source: §F.3 · pp. 31–32

Related papers

Loading 2609.32590v1…