PaperlayerView exact revision on arXiv

Source-linked AI summary

Wontopos Tablet 2: Measuring Multilingual and Multimodal Memory Retrieval Without Lexical Matching

Sunwoo Kim

arXiv:2608.23920v1cs.IRcs.CLcs.CV

TL;DR

Evaluating long-term memory across text and captionless-image retrieval requires separating retrieval quality from reader effects and testing language independence directly. This paper measures tablet-2 with controlled lexical and dense-baseline comparisons, finding strong multilingual retrieval alongside substantial sensitivity to readers, re-asking, language, captions, and configuration.

  • Problem

    Evidence for long-term memory retrieval is limited by reader-dependent benchmark scores and insufficient tests of language independence, especially for text-free photographs.

  • Method

    The paper evaluates tablet-2 on text benchmarks and cross-lingual captionless-photograph retrieval, using controlled reader, re-ask, lexical, and dense-baseline comparisons.

  • Results

    Across text and multimodal evaluations, tablet-2 performs strongly but results vary materially with evaluation and retrieval conditions, including 91.4% recall@5 against BM25’s zero on captionless photographs.

  • Takeaways & Limitations

    The comparisons support treating tablet-2’s benchmark placement as conditional evidence rather than a universal ranking, while exposing concrete multilingual and product-level weaknesses.

  • Takeaways & Limitations

    The reader is not controlled across systems, and reader and re-ask choices move scores by 2.0 and 8.9 points, respectively.

Abstract

from arXiv · show

We measure tablet-2, a production long-term memory engine for language models, on the text benchmarks the field already uses and on cross-lingual retrieval of photographs stored with no text at all. Its retrieval path contains no lexical matching, no keyword scoring, and no language model of its own. On LongMemEval-S (500 questions) it scores 95.7% [93.4, 97.1]; on BEAM-1M (700 questions, 2.21M stored memories) 67.5% [64.8, 70.2]. Those are question-sampling intervals, not the run-to-run spread, which is an order of magnitude narrower. Most of the paper is about how little they mean alone. Holding engine, corpus, settings and judge fixed, changing only the reader moves LongMemEval-S by 2.0 points; changing only the re-ask budget moves BEAM-1M by 8.9. Neither is stated in the reports we compare against, and the second exceeds most gaps there, so we give that table as a placement and not a ranking. For the multimodal axis we run two controls. Against BM25, configured as strongly as we could, we reach 95.2% mean recall@5 over 70 store-and-query language cells where BM25 reaches 19.0% and is exactly zero in 54. On captionless photographs a lexical method has no document to score at all. Open dense baselines on 300 Crossmodal-3600 photographs in 14 languages show that density confers no language independence: one scores 91.0% on English and 4.7% on Russian from identical image vectors, and a multilingual variant collapses on Telugu and Swahili. Our spread across languages is 14.0 against their 27.5 and 27.7. Three results run against us and are reported at equal weight: low-resource languages degrade sharply (Swahili 53.0%, Telugu 64.0%), attaching captions lowers cross-lingual retrieval by 11.4 points, and one setting omitted into one stage of our own retrieval cost 37 points of Korean top-1 accuracy while leaving nine languages untouched.

WONTOPOS

The section emphasizes that retrieval quality is best assessed with the reader held constant, while deployment details can materially affect reported performance. A single retrieval at medium reader effort scores 93.8 and delivers fewer memories than a stronger reader using three passes.

  • Deployment failure analysis: 37 points of Korean top-1 accuracy were lost on short queries when one setting was omitted from one retrieval stage, while nine other languages were unaffected.The authors initially misattributed this limitation to the stage itself before separating the cases.
  • Retrieval measurement: 93.8 is the engine’s score with re-ask off, medium reader effort, and one retrieval pass.This setting edges the stronger reader’s 93.7 with three retrieval passes.
  • Retrieval measurement: 30% fewer memories are delivered by the single-retrieval setting than by the stronger reader using three retrieval passes.The comparison is presented as the paper’s closest measurement of retrieval quality with the reader held still.

1 Introduction

The paper evaluates production tablet-2 on text benchmarks and cross-lingual retrieval of captionless photographs, separating memory retrieval from downstream reading and judging. It emphasizes that benchmark scores depend substantially on unstated evaluation choices, while acknowledging closed-source and replication limits.

  • Motivation: A memory system should return relevant memories, whereas reasoning and answer phrasing belong to separate downstream components.Scoring the language model’s answer measures the memory system and reader together rather than retrieval alone.
  • Text benchmarks: 95.7% and 67.5% are reported on LongMemEval-S and BEAM-1M, using 500 and 700 questions respectively.BEAM-1M uses a corpus three orders of magnitude larger than the other benchmark; the paper also varies reader, re-ask budget, and context expansion.
  • Evaluation sensitivity: 8.9 points is the measured score swing from changing re-ask budget alone, while reader and context choices also move results substantially.Because such settings are often unstated, published score comparisons are treated cautiously.
  • Multilingual multimodal retrieval: tablet-2’s retrieval path uses no BM25, keyword or token-overlap scoring, or language model, and tests language independence with captionless photographs.The multimodal evaluation includes a lexical control where retrieval is structurally undefined and dense baselines testing whether density alone confers language independence.
  • Negative results: Low-resource languages degrade sharply, and captions harm cross-lingual photo retrieval, both reported as negative results rather than benchmark artifacts.The paper presents these defects at the same weight as its positive results.
  • Scope and reproducibility: tablet-2 is closed source, so the paper reports observable behavior and caller-configurable settings rather than its retrieval architecture, ranking function, or storage layout.The authors distinguish limited rebuildability from checkable measurements produced through the ordinary request interface.

2 System under measurement

tablet-2 is a store-and-retrieve engine whose interpretation is supplied by a caller or separate reader, while multimodal evaluation directly tests returning photographs for text queries. Its commercial retrieval implementation is not disclosed, but the paper exposes caller-facing controls and separates retrieval stages for measurement.

  • Measurement boundary: tablet-2 returns memories without reasoning over them; text scores therefore measure the engine together with the language-model reader, whereas multimodal tests use no reader.Interpretation is the caller’s or a separate component’s responsibility.
  • Multimodal retrieval: Captionless photographs can be retrieved by text descriptions because tablet-2 indexes photographs and text for cross-modal access, even when the stored photograph has no textual record.The tested direction is a photograph stored with no caption, title, alt text, or surrounding message.
  • Scope and limitation: The commercial retrieval implementation is undisclosed, so the paper reports product behavior rather than construction.This limitation concerns how the engine works internally, not the reported measurements.
  • System structure: The retrieval path has multiple stages, including a separable photograph-description matching stage that can be measured independently from surrounding handling.This structure lets the paper distinguish weakness in matching from weakness in the rest of the retrieval path.
  • Caller-facing controls: Re-ask permits 0–3 additional retrieval passes, while context expansion attaches neighboring material and image budget allows 1–5 photographs per response.The default image budget is 1, the maximum is 5, and recall@5 is measured against that five-image ceiling.
  • Re-ask protocol: Re-ask is a caller-controlled ceiling rather than a fixed schedule: later passes exclude previously returned memories, stop when nothing new appears, and never invoke a language model.The budget is spent differently across benchmarks because the caller’s loop decides whether to continue, while the engine itself remains stateless between passes.

3 Benchmark 1: LongMemEval-S

LongMemEval-S evaluates retrieval over 500 multi-session-dialogue questions with binary judgments from gpt-4o at temperature 0. Controlled experiments show that reader choice and re-asking materially affect results, while cross-generation gains and benchmark scores require cautious interpretation because intervals overlap and campaigns differ.

  • Evaluation protocol: 500 questions are scored by a judge model using only yes/no verdicts, with gpt-4o at temperature 0 as the unchanged benchmark default.Answer generation and judging are separate processes, so no model grades its own output.
  • Reader effects: 2.0 points separate the two readers when engine, corpus, retrieval settings and judge are held constant.The lowest Opus 5 run was 95.2, above the highest GPT-5.6-sol run at 94.0; both configurations had a run-to-run spread of 0.8.
  • Re-asking effects: 1.2 points are added by re-asking, raising the score from 93.8 to 95.0 under a controlled one-retrieval comparison.The reader is not merely reasoning over fixed input: the stronger reader requested more than twice as often and answered from 40% more material.
  • Re-asking effects: 4.2 points are lost on single-session-user when re-asking is enabled, despite gains in four of the other five categories.Re-asking can add material that competes with evidence already present in the first retrieval.
  • Cross-generation comparison: 1.4 points separate tablet-2 from scroll-1.2 with the same reader, but overlapping intervals and separate campaigns prevent treating this as an established difference.The reported intervals are [89.7, 94.4] and [91.1, 95.4].

4 Benchmark 2: BEAM-1M

On BEAM-1M, tablet-2 is evaluated on 700 questions over 2,212,504 stored memories, with results shaped substantially by re-asking, reader and judge choices, and context-delivery policy. The benchmark also exposes measurement limitations, including event-ordering rubric mismatch and corpus-ingest defects.

  • Benchmark setup: 700 questions span 35 conversations and 2,212,504 stored memories across ten question types.The corpus contains 74,630 turns, and the types range from preference following to event ordering.
  • Comparability: 7.5 points is the gap between the top published system and ours, while reader changes are worth 2.0 points on LongMemEval-S and re-ask changes 8.9 points on BEAM.Published rows differ in readers, judges, prompts, intervals, and retrieval budgets, so the table cannot be read as a ranking.
  • Re-ask sensitivity: 8.9 points is the value of re-asking: disabling it lowers BEAM from 67.5% to 58.6%.The same setting is worth 1.2 points on LongMemEval-S, indicating a larger effect on the 2.2M-memory BEAM corpus.
  • Context expansion: 70.3% is configuration C’s score, but it requires 16,590 tokens per query versus 2,484 for B, delivering 6.7× the context for 2.8 points.Because expansion can worsen temporal_reasoning, knowledge_update, and contradiction_resolution, C is reported as preliminary rather than as the main result.
  • By-type results: 23.6% is the event_ordering score, depressed by a mismatch between the rubric’s topic-level items and the answers’ event-level items.The authors decline to reshape answers to the rubric because the benchmark specifies ten items but not their unit of granularity.
  • Corpus limitations: 0.3% of turns were truncated, and one turn failed to store, leaving both harness defects in the measured corpus.The truncated turns exceeded 8,000 characters; the failed turn was one of 74,630.

5 Benchmark 3: multilingual retrieval of photographs

The photograph benchmark shows that lexical and dense retrieval are not inherently language-agnostic, while tablet-2 remains substantially more consistent across query languages. It also exposes limitations: captions can displace captionless results, low-resource languages remain weakest, and corpus size dominates run-to-run uncertainty.

  • Dense baselines: 91.0% on English versus 4.7% on Russian for the English text tower shows that identical image vectors do not ensure language independence.The multilingual tower raises its mean from 19.4% to 68.5% but falls to 5.0% on Telugu and 6.0% on Swahili.
  • Dense baselines: 14.0 versus 27.5 and 27.7 is the cross-language standard-deviation comparison favoring tablet-2’s consistency, not a level-ranking claim.The baselines also lose 90.0 and 83.3 points between their best and worst languages.
  • Caption effects: 11.4 points is the cross-lingual retrieval decrease after attaching English captions, as one language gains three points while thirteen lose an average of eleven.In mixed stores, captioned photographs can occupy the five result slots and push captionless answers out; this remains an unfixed product defect.
  • Measurement uncertainty: 86.7, 83.3 and 90.0 recall@5 across three re-ingests have σ = 2.7, while the single-measurement Wilson 95% interval spans [70.3, 94.7].At n = 30, uncertainty is dominated by corpus size rather than run-to-run variation; 300-photograph measurements have intervals of roughly ±4 points.

6 Attribution: whose limit is a weak language?

The section separates weak-language limits caused by a retrieval stage from losses introduced by surrounding handling. Korean exposed a fixable 37-point handling gap, whereas real-photograph language differences remained intrinsic to the stage.

  • Korean case: +1.3 points at rank 1 on real-photograph captions, rather than +37, shows the request-setting effect is confined to short queries.Long human-written captions carry enough signal that the missing field does not reorder the top of the list substantially.
  • Real-photograph case: 36.7% to 73.3% Hindi recall@5 after a stage-version change shows that component quality can move the entire language profile.Stage-alone exhaustive scoring is a ceiling for systems built on that stage, not a baseline; tablet-2 reaches it without examining every record.
  • Real-photograph case: 47% Swahili recall@5 was reproduced by the stage alone, with every real-photograph language agreeing within 0.7 points between stage and full service.This opposite verdict assigns the language profile to the stage rather than to the surrounding retrieval path.

7 Limitations

The evaluation is limited by self-reported comparisons, model-judge and reader variability, rank-five and corpus-size ceilings, and non-like-for-like multilingual comparisons. Known weaknesses include low-resource degradation, caption-related cross-lingual loss, configuration errors, and harness truncation.

  • Evaluation comparability: Every number and comparison is self-reported, and default model judges are not neutral instruments.Judges are sensitive to candidate presentation, although these protocols compare one answer against a reference rather than placing two answers side by side.
  • Evaluation comparability: 2.0 points from the reader model on LongMemEval-S and 8.9 from the re-ask budget on BEAM-1M are lower bounds on comparison movement.The reader is not controlled across systems, making this the largest single caveat on the text results.
  • Multimodal measurement: Recall@5 is capped at five photographs, making rank six indistinguishable from never finding the correct photograph.“Not in the top five” is the floor these multimodal experiments can observe.
  • Multimodal measurement: σ = 2.7 points across three re-ingests contrasts with about ±12 for the binomial interval at n = 30, while 300-photograph measurements are roughly ±4.The synthetic corpus has only 30 images, and per-cell intervals are provided for the larger measurements.
  • Multilingual limitations: 53.0% Swahili and 64.0% Telugu recall@5 are the system’s worst results, so callers serving these languages should size expectations by them rather than the mean.The low-resource weakness is shared, but it remains a deficiency.
  • Known defects and multilingual limitations: 11.4 points is the average cross-lingual retrieval degradation caused by attaching captions, while 19 BEAM questions errored and 211 long turns were truncated.One additional turn failed to store; XM3600 captions are independently written, so cross-language comparisons also mix retrieval with annotator choices.

8 Reproducibility

The paper releases a reproducible harness and supporting scripts that record retrieval, reading, judging, and per-question outcomes. It also documents protocol-specific grading, implementation checks, benchmark records, and the independent BM25 control.

  • Released harness: The released harness records each question, received context, retrieval endpoint, reader, judge, and configuration, while rejecting apparent provider-error runs.It counts tokens only when given a tokenizer and does not estimate them.
  • Grading protocols: LongMemEval uses binary grading, whereas BEAM-1M averages three-point rubric items and scores ordering questions by rank correlation.Both grading protocols are included in the harness, and using the wrong protocol changes BEAM grading.
  • Implementation validation: 400 random tied-input checks found no disagreement between the plain-Python rank correlation and the standard-library implementation, while released aggregation reproduced five runs within 0.05 points.The harness runs without third-party packages.
  • Artifacts and records: The harness, corpus scripts, and per-question records for every reported configuration were published on 29 August 2026, including two unreproduced configurations.The two additional records are the re-ask-off run and an expansion run that lost nineteen questions to errors.
  • Baseline implementation: The BM25 control is independent of the engine, using Okapi BM25 with k1 = 1.5 and b = 0.75 over caption strings in plain Python.It uses no third-party imports.
  • Benchmark records: LongMemEval-S records include answers, verdicts, passes, delivered memories, supporting-evidence status, and benchmark question identifiers for all eight runs of Table 2.The supporting-evidence field is used to distinguish retrieval from reader performance.

9 Related work

Related work situates tablet-2 among memory systems, lexical and dense retrieval, multilingual retrieval, image-text retrieval, and iterative retrieval with language-model grading. Unlike broader memory managers and iterative retrieval methods, tablet-2 is described as a narrower store-and-return system whose re-ask protocol keeps models out of the retrieval loop.

  • Memory systems and the benchmarks built for them: tablet-2 stores and returns memories without deciding what a memory means, unlike MemGPT, MemoryBank, and Mem0.MemGPT pages material between working context and external storage; MemoryBank adds forgetting; Mem0 extracts and consolidates facts during conversation.
  • Retrieval with and without term matching: BM25 is the lexical control, while tablet-2 contains no lexical scoring and is situated against supervised and unsupervised dense retrievers.BEIR and MTEB helped make lexical-versus-dense retrieval comparisons routine.
  • Retrieval in many languages: Multilingual retrieval work established evaluation across many languages, including settings where retrieval returns nothing relevant.Mr. TyDi and MIRACL measure multilingual passage retrieval, while NoMIRACL examines empty-retrieval cases.
  • Finding a picture from a sentence: Contrastive image-text pretraining enabled sentence-to-photograph retrieval, with shared image encoders isolating differences caused by English versus multilingual text encoders.The controls pair one model’s image encoder with its English text encoder and a multilingual encoder distilled from it.
  • Searching more than once, and grading the result: The re-ask protocol differs from iterative retrieval by refusing to put a model in the loop, while both benchmarks use language-model grading.The grading paradigm follows surveyed language-model evaluation and is subject to documented sensitivities.

10 Conclusion

The conclusion frames tablet-2’s contribution as measuring both situational retrieval performance and retrieval independence from language and text, while treating benchmark scores as context-dependent rather than rankings. It emphasizes bounded multimodal comparisons, substantial weaknesses in low-resource and captioned settings, and independent validation of reported measurements.

  • Benchmark performance: 67.5% on BEAM-1M and 95.7% on LongMemEval-S are reported with 95% question-sampling intervals of [64.8, 70.2] and [93.4, 97.1].The authors caution that BEAM-1M placement beside published numbers is not a ranking because reader choice changes results by 2.0 points and re-ask budget by up to 8.9.
  • Language and text independence: 91.0% on English versus 4.7% on Russian shows that open dense retrieval is not uniformly language-independent, while captionless photographs leave lexical methods with nothing to score.The conclusion presents the multimodal comparison as bounded by an upper dense-retrieval baseline and a lower lexical floor.
  • Limitations: 53.0% on Swahili and 64.0% on Telugu are tablet-2’s worst results, and attaching captions reduces cross-lingual photograph retrieval by 11.4 points on average.The open baselines reach 6.0% and 5.0% on Swahili and Telugu, showing that the difficulty is not unique to tablet-2 without making its results good.
  • Diagnostic procedure: Isolating the stage that decides a weak language or condition distinguishes a limitation in that stage from a limitation in surrounding handling.The authors ran this diagnostic twice and obtained different verdicts, making it the procedure they most want to pass on.
  • Measurement validation: Three published numbers were wrong because of contaminated latency data, an uncounted delivered-token figure, and an omitted error denominator, and none was caught by rerunning the same analysis.They were caught by measuring the same thing a second way.

A Prompts

The appendix gives the exact reader and judge prompts used for each benchmark, preserving reproducibility while applying benchmark-specific answer constraints and strict grading procedures. It also clarifies that prompts were not benchmark-tuned and ambiguous judge outputs were excluded.

  • A Prompts: Exact prompts are reproduced verbatim from the sources that produced the runs, and LongMemEval-S reader and judge prompts are published on the model pages.This supports reproducing the reported scores without paraphrase drift.
  • A.1 BEAM-1M reader prompt: BEAM-1M uses the same reader prompt for every run, restricting answers to the supplied memories and requiring explicit uncertainty when the answer is absent.The prompt also prohibits outside knowledge.
  • A.2 BEAM-1M judge prompt: The BEAM-1M judge uses the benchmark repository’s unified_llm_judge_base_prompt verbatim, without modification, at temperature 0.The prompt is reproduced by reference to avoid transcription drift.
  • A.3 LongMemEval-S reader prompt: LongMemEval-S requires answers based only on retrieved memories, computes elapsed time from the asking date, and reports conflicting values with their relative recency.These rules use memory dates and the question’s asking date rather than an external current date.
  • A.3 LongMemEval-S reader prompt: LongMemEval-S advice answers must reflect remembered preferences and choices, while factual answers remain concise, use the question’s language, and admit when information is missing.The prompt is longer than BEAM-1M because LongMemEval-S requires relative-time arithmetic and conflict handling; neither prompt was rubric-tuned.
  • A.4 LongMemEval-S judge prompt: LongMemEval-S judges run separately at temperature 0 and return only yes or no under each benchmark category’s unchanged rule.The judge receives the question, correct answer, and model response, then evaluates whether the response is correct.
  • A.4 LongMemEval-S judge prompt: Ambiguous verdicts leave questions ungraded and outside the denominator rather than being resolved by substring matching.Interpreting “the answer is not yes” as yes would raise every affected score.
Loading 2608.23920v1…