Source-linked AI summary
Good Memory Has ECC: Evaluating the Memory of Vision-Language Models Beyond Accuracy
Shmuel Berman, Jia Deng
TL;DR
Memory benchmarks often rely on accuracy over long text or video, but accuracy alone does not capture efficient retrieval, compression, or cost-sensitive uncertainty. ECCBench evaluates these properties and finds that VLMs compress text but generally not video, while several non-Transformer backbones have better compression-calibration tradeoffs than RoPE Transformers.
Problem
Existing memory benchmarks mainly measure accuracy on long text or video, leaving efficiency, compression, and calibration insufficiently evaluated.
Method
ECCBench measures memory through computation required per answer, accuracy gains on structured inputs, and abstention under explicit error costs.
Results
VLMs compress text but generally not video and are poorly calibrated across both modalities; several non-Transformer architectures have better compression-calibration tradeoffs than RoPE Transformers.
Takeaways & Limitations
Memory quality should be assessed beyond aggregate accuracy, including efficiency, compression, and calibration, and architecture can substantially affect these properties.
Takeaways & Limitations
Compression is difficult to measure for models near ceiling, and natural-video compression claims are only qualitative because the underlying distributions and entropy are unknown.
Abstract
from arXiv · showhide
Memory is widely viewed as an important unsolved problem for LLMs and VLMs, and current benchmarks typically evaluate it by testing accuracy over long text or video. However, accuracy alone misses properties that matter for real long-horizon tasks. We introduce ECCBench, a benchmark and evaluation protocol that measures memory beyond a system's capacity--its raw accuracy at a specific budget--via three axes we call ECC: efficiency--the computation, in FLOPs, needed to answer from memory; compression--whether compressible inputs are remembered more accurately or efficiently; and calibration--whether the system abstains in response to its own uncertainty and the cost of an error. We find that pretrained VLMs compress their memory over text but not video and are poorly calibrated on both. Among a broader set of memory backbones, several non-Transformer architectures achieve better compression-calibration tradeoffs than RoPE Transformers, suggesting they may be useful components for agents operating over long horizons.
1 Introduction
The paper argues that standard accuracy-based memory evaluation is insufficient. Strong memory also requires efficient retrieval and compression-aware evaluation that avoids confounding memory with question or data biases.
- Motivation: Accuracy alone does not establish that an LLM or VLM has good memory.Existing benchmarks typically ask questions about long text or video and judge memory through answer accuracy.
- Efficiency: Efficiency matters because a memoryless system can achieve high accuracy by spending unlimited computation to search the original input.The paper defines good memory as fast and effortless recall rather than brute-force rechecking.
- Compression: Question selection can let systems exploit data biases instead of remembering broadly.If every question concerns rare faces in a video, remembering faces alone can outperform a system that retains other information.
- Compression: Randomly querying all information makes useful performance depend on compression of redundancy in the input.With finite storage and sufficiently long inputs, selective retention cannot help when every piece of information is equally likely to be queried.
Strong Memory Means ECC
ECCBench evaluates memory beyond accuracy through efficiency, compression, and calibration. It exposes weaknesses in VLM memory and shows that several non-Transformer backbones have better compression-calibration tradeoffs than RoPE Transformers.
- ECC axes: Efficiency is compute spent per query, compression is benefit from input structure, and calibration is abstention when recall is uncertain and errors are costly.These axes distinguish systems that have similar standard recall accuracy.
- Evaluation design: Existing evaluations confound compression with saliency because task- or domain-dependent priors can determine what information systems retain.Adversarially chosen questions can make saliency useful or counterproductive, obscuring whether performance comes from compression.
- Calibration: Finite memory makes calibration necessary: systems should abstain when requested information was not preserved or uncertainty makes errors costly.Most existing benchmarks omit both an abstention option and an explicit cost for incorrect answers.
- ECCBench: ECCBench measures efficiency, compression, and calibration alongside memory accuracy.It measures computation per answer, accuracy gains from structured inputs, and abstention under uncertainty and error costs.
- VLM findings: VLMs compress text but generally not video and remain poorly calibrated across both modalities.Increasing the cost of an incorrect answer has little effect on their coverage.
- Backbone comparison: Several non-Transformer architectures achieve better compression-calibration tradeoffs than RoPE Transformers.The experiments indicate that architecture changes accuracy, compression, and calibration properties of memory.
2 Related Work
Prior work evaluates memory through long-context and long-video accuracy, formal memory mechanisms, associative recall, compression, and memory architectures. ECCBench builds on these traditions while separating memory properties from saliency and reasoning demands.
- Long-context and long-video benchmarks: Long-context and long-video benchmarks test memory with questions about documents or videos, but confound memory with predicting what matters.Natural and synthetic tasks include question answering, summarization, retrieval, ordering, and counting.
- Definitions and evaluations of memory: Prior memory definitions span readable and writable states, episodic, semantic, and procedural memory, and model components treated as associative memories.The paper abstracts over these mechanisms rather than adopting one architectural definition.
- Associative recall and compression: ECCBench uses associative recall because it requires little semantic knowledge or multistep reasoning, allowing other memory properties to be assessed independently.Its behavioral compression notion differs from prior storage-oriented compression methods.
- Memory architectures and agents: Memory architectures and agents retain long histories through recurrent state, learned updates, reduced attention caches, or searchable and editable stores.These approaches include memory-augmented models and agents with internal or external memory.
3 Background
The paper formalizes memory as an online write-read system and evaluates capacity, efficiency, compression, and selective calibration. These properties are defined using recall accuracy, computation, input regularity, coverage, and accuracy on answered queries.
- Memory model: A memory system observes a stream, updates an internal state, and later answers questions about previously seen observations.Inputs may arrive piecemeal, questions may occur at any time, and stream length is not revealed in advance.
- Capacity: Capacity is correct-answer rate on maximum-entropy inputs at fixed stream length and FLOPs budget.Comparing capacity across stream lengths measures how much incompressible input the system can remember.
- Efficiency: Efficiency is the smallest total FLOPs needed to reach a target correct-answer rate.The total includes computation for reading the stream and answering the query.
- Compression: Compression is accuracy gained on equally long streams with greater regularity, or alternatively FLOPs saved at the same accuracy.Regularity is compared across stream distributions while holding length, budget, and query distribution fixed.
- Compression and saliency: Compression exploits regularity in the stream, whereas saliency exploits knowledge of which questions are likely to be asked.The distinction is formalized through the information needed to preserve answer-relevant differences between streams.
- Calibration: Calibration measures whether a system answers when memory is reliable and abstains when error risk is high.The paper evaluates coverage and accuracy-on-answered across reward and cost profiles, including responsiveness to changing error costs.
4 Experiments
ECCBench evaluates memory through controlled text and video streams, natural videos, compute-aware recall, abstention under costs, and diverse memory backbones. Results show text compression but weak video compression and calibration in VLMs, while several non-Transformer backbones outperform RoPE Transformers on memory tradeoffs.
- Experimental setup: ECCBench evaluates nine pretrained VLM families on controlled-entropy text and video, natural videos, and a RAG-like memory system using compute-aware scoring.Arithmetic cost includes stream reading and answering, with token counts used for VLM answer computation.
- Text: Only three VLMs exceed 60% accuracy on high-entropy text, despite most being highly accurate on low-entropy text.The benchmark defines above chance as at least 60% accuracy against a 50% chance rate.
- Text: VLMs exploit text regularity mainly through higher accuracy rather than lower compute, with only InternVL3.5 (38B) reaching similar accuracy at fewer FLOPs on low-entropy data.Other models fail to maintain 60%+ regular-data accuracy on irregular data regardless of tested compute.
- Video: Synthetic video memory is weak: most models remain near chance across entropy levels, and only InternVL3.5 Thinking (30B, A3B) exceeds a ten-point compression gap.The retrieval-based system compresses at L = 64, while average capacity is lower than for text.
- Video: Natural-video compression is also weak, with the mean balanced-accuracy contrast reaching only +1.34 points after correcting label-prior effects.Only InternVL3.5 (8B) shows strong compression, and some models exhibit anti-compression.
- Calibration: Increasing error cost barely changes VLM coverage, and many models obtain lower payoff than abstaining on every question.Models generally adjust behavior toward the payoff scheme, but improvements remain small and selectivity is imperfect.
- Alternative backbones: Several non-Transformer architectures achieve better compression-calibration tradeoffs than RoPE Transformers, which have low capacity and weak accuracy gains when coverage is reduced.RetNet and TTT-MLP show roughly 30% gaps between peak low-entropy and max-entropy accuracy; near-ceiling accuracy can make compression differences difficult to measure.
5 Conclusion
The paper evaluates memory beyond accuracy through Efficiency, Compression, and Calibration, finding weaknesses in VLM memory and advantages for several non-Transformer backbones on longer sequences.
- ECCBench measures memory beyond accuracy through Efficiency, Compression, and Calibration.The evaluation is designed for VLMs, retrieval-augmented models, and other intelligent systems.
- VLMs compress text but remain weak compressors on video, while showing limited capacity and poor adjustment of abstention rates to explicit costs.
- Several non-Transformer architectures handle sequences beyond their training length better than RoPE Transformers.
A Benchmark Details
ECCBench constructs entropy-controlled symbolic streams and membership queries to test memory across input lengths and modalities. Its generator uses variable-order n-gram rules and distinguishes low-entropy from maximum-entropy streams.
- Entropy-controlled generator: Streams use a fixed alphabet of 16 symbols and variable-order n-gram rule sets with context lengths from 1 to 8.
- Entropy bands: Low-entropy streams average 0.67 bits/symbol, while max-entropy streams are fixed at 4 bits/symbol.
- Stream generation: The generator resamples rule sets until each realized stream falls within its target empirical Lempel–Ziv entropy band.
- Queries: Each length-and-band cell contains 20 sequences paired with 16 membership queries using three- to six-symbol candidates.
- Queries: Positive candidates are copied contiguous subsequences, whereas negatives are uniformly sampled candidates absent from the stream.
A.2 EasyHuman
EasyHuman tests whether models remember rare deviations in highly regular patterns across text and video. The split uses repeated motifs, deviation-focused queries, and matched video renderings with sequential and spatial variants.
- Motivation: EasyHuman targets the human-associated ability to remember rare events that break predictable patterns.The authors did not run a human study despite naming the split EasyHuman.
- Results: Models partly compress text but do not compress video, with video accuracy near chance despite extreme input redundancy.
- Results: Frontier models exceed 90% on EasyHuman text at both lengths, yet none is distinguishable from chance on video.
- Construction: Streams repeat motifs of 3–5 symbols with one or two deviations, and queries target deviations and their corruptions.
- Video rendering: Video renders each symbol as one letter on one of three conveyor belts at 448 × 448 pixels and 1 frame per second.
- Video queries: Sequential and spatial variants differ in whether incorrect candidates preserve temporal order or alter belt placement while retaining glyphs.
A.4 Dataset documentation
The ECCBench release includes Croissant metadata, dataset manifests, and documentation covering standard datasheet and Responsible-AI categories.
- Documentation: The HuggingFace release includes Croissant 1.0 metadata covering standard Datasheets-for-Datasets and Responsible-AI fields.
- Release contents: The release provides a questions index and per-bucket manifests needed by the codebase.
B Pretrained VLMs Experimental Details
The VLM experiments use fixed prompts for text, synthetic video, and natural-video questions, with explicit answer and abstention formats. The protocol documents FLOPs measurement, natural-video evaluation, calibration, and substantial GPU usage.
- Evaluation protocol: The VLM protocol includes prompt specifications, analytical FLOPs measurement, natural-video evaluation, and a two-arm calibration protocol.These components are documented across the experiment and appendix sections.
- Prompt design: Models answer yes, no, or abstain using bracketed tokens, while natural-video questions use an uncertainty token.Responses without an appropriate bracketed answer within the allowed budget are treated as implicit abstentions.
- Video setup: Synthetic video prompts describe letters moving sequentially down fixed conveyor belts, with one letter visible at a time.The rendered videos show each letter in a single frame, so lane progression is conveyed across frames rather than within-frame motion.
- Prompt design: The evaluation uses fixed prompts for text, synthetic video, and natural-video questions, with separate instructions for sequence matching and video-event comparison.Text questions test whether a candidate appears as a contiguous subsequence; video questions ask whether events match in order and conveyor-belt assignment.
- Compute: Experiments used A40, H200, L40, and RTX 3090 GPUs, with approximately 60,000 GPU-hours consumed overall.Inference jobs used 2–7 GPUs depending on model size, while backbone training used single-GPU nodes.
B.3 Per-model FLOPs equations
Per-model FLOPs are computed analytically from input, output, and architectural shapes rather than profiled measurements. The accounting covers vision encoding, connectors, language-model reading and decoding, and memory write/read costs.
- Accounting principles: FLOPs are computed analytically from query shape and architectural constants, with no profiler measurements or fitted quantities.The aim is a like-for-like arithmetic comparison across architectures.
- End-to-end cost: The total cost includes vision encoding, connector computation, language-model prefill, and decoding, with generation reusing cached keys and values.Reading the prompt produces the first generated-token distribution, so later tokens incur layer and cache-attention costs.
- Vision encoder: The vision cost accounts for patch processing, attention span, feed-forward operations, and model-specific frame grouping.Fixed spans make the vision term linear in the number of video frames for every evaluated model.
- End-to-end cost: Reading generally dominates decoding, especially for video, because many more input tokens are processed than output tokens.The reported costs distinguish millions, billions, and trillions of operations across VLMs.
- Baseline: The short-stream perception check uses L=8 as each model’s baseline and interprets performance drops at longer lengths as memory loss.Cohort averages at L=8 were 86.8% on text, 71.0% on one video variant, and 65.1% on the other.
- Compression analysis: Natural-video compression is measured as low-density accuracy minus high-density accuracy, with balanced accuracy used to account for differing label priors.The mean contrast changes from +0.45 to +1.34 points after equal weighting, and InternVL3.5 (8B) reaches +19.0 at L=128.
B.7 RAG-Pipeline for Video Retrieval
The video retrieval pipeline embeds overlapping windows, retrieves the most relevant windows, merges them, and sends the resulting clip to Qwen3-Omni. It can reduce natural-video answering cost substantially, but retrieval errors and weak rejection hurt synthetic-video accuracy and calibration.
- Pipeline: The pipeline embeds 16-frame windows at stride 4, retrieves top-ranked windows, merges overlaps, and passes the composed clip to Qwen3-Omni.At lengths of at least 128, it retrieves the top three windows.
- Pipeline cost: Embedding one 16-frame window costs 32.9 trillion operations, while query embedding costs 0.56 trillion on natural video and 11 trillion on synthetic video.The index cost is amortized across questions in the reported per-answer comparison.
- Results: 4.7×–42.5× lower per-answer cost is achieved on natural video once indexing is amortized, while accuracy matches or exceeds the full-video baseline.Savings are larger for longer videos because the model reads retrieved windows instead of the full stream.
- Results: Synthetic-video accuracy is 3.4 points below the full-video baseline because retrieval becomes the bottleneck when many windows look similar.For yes-questions, accuracy closely tracks the retriever hit rate; no-questions degrade from 64.9% at L=64 to 46.8% at L=512.
- Calibration: The calibration evaluation varies the reward for correct answers and penalty for wrong answers while permitting explicit abstention.Arm A uses +3/-1 with a 25% confidence threshold, whereas arm B uses +1/-4 with an 80% threshold.
- Calibration: Calibration analyses count context overflow, format non-compliance, and missing answers as abstentions in addition to explicit abstention outputs.Coverage is the fraction answered, while accuracy is computed over answered items only.
C.1 Configurations and training
The backbone study trains and evaluates diverse recurrent, attention-based, and hybrid memory architectures under a common synthetic membership task and FLOPs convention. It excludes a no-positional-encoding Transformer that stayed at chance and compares models beyond their training lengths.
- Architectures: A Transformer without positional encoding stayed at chance on the training distribution and was therefore not reported.A Transformer with a small convolution was used instead to evaluate the effects of RoPE.
- Training: All architectures are three-layer memory backbones trained for up to 36 hours, with the best of four configuration runs selected for evaluation.Training uses sequence lengths 32–1024 to simulate streaming memory available at every timestep.
- Training data: The shared synthetic dataset contains 500,000 randomly generated sequences, 32 queries per sequence, and a 16-symbol vocabulary.Validation uses 5% of examples, with architecture-dependent batch sizes and learning rates.
- FLOPs accounting: The backbone FLOPs convention counts matrix multiplications while omitting elementwise operations such as normalization, RoPE, activation, and softmax.The backbone accounting is kept separate from the VLM accounting.
- Architectures: Evaluation compares recurrent, gated-attention, DeltaNet, RWKV, DeltaFormer, TTT-MLP, MoM, Memory Mosaic, Compressive Transformer, and GRU-style backbones.The architectures differ in their recurrent states, attention mechanisms, memory updates, and feed-forward components.
- Memory mechanisms: The memory architectures include explicit mechanisms such as cached sliding-window attention, fast-weight updates, routed memories, compressed memory slots, and recurrent hidden states.These mechanisms are represented in the per-layer formulas used for the architecture comparisons.
D Limitations and Broader Impact
The study’s scope is limited by its synthetic, associative-recognition focus and several measurement and scaling assumptions. It nevertheless aims to encourage more computationally efficient and uncertainty-aware memory systems while acknowledging possible beneficial and harmful impacts.
- Scope: ECCBench omits natural text and evaluates associative recognition rather than free-form recall, temporal reasoning, interference, or complex reasoning from memory.The experiments do not establish that ECC findings transfer to these untested behaviors.
- Measurement assumptions: Efficiency is measured with arithmetic FLOPs under one query per stream, excluding factors such as memory bandwidth and communication costs.Multi-query workloads may rank systems differently when expensive writes can be amortized across later queries.
- Synthetic data: Synthetic experiments control for other skills and priors but still depend on models extracting and retaining the target information effectively.Model priors may make some glyphs more compact, inflating capacity scores.
- Measurement assumptions: Natural-video compressibility is assessed with a heuristic validated only through sanity checks because no established feature-space measure exists.The measure was chosen in the absence of an established alternative operating on feature rather than pixel space.
- Reporting scope: Complete results are unavailable for closed-source models because their FLOP usage cannot be accessed, although ECC’s definitions and axes still apply.This limits direct efficiency reporting for those systems.
- Broader impact: The broader-impact goal is to promote resource-efficient systems that withhold answers under uncertainty, while better memory could amplify both beneficial and harmful applications.The framework is presented as a way to compare memory across differing computational, certainty, and accuracy requirements.