PaperlayerView exact revision on arXiv

Source-linked AI summary

MemEvo: Automatic Discovery of Streaming Video Memory Mechanisms

Guohong Liu, Jialei Ye, Shanhui Zhao, Yunxin Liu, Yuanchun Li

arXiv:2609.36581v1cs.AI

TL;DR

Streaming video systems must compress an unbounded visual history into bounded memory before future queries are known, yet memory design involves coupled choices whose effects emerge through downstream execution. MemEvo searches executable memory programs with an LLM proposal model and fixed evaluation pipeline, improving frozen backbones across benchmarks while maintaining bounded memory and efficient inference.

  • Problem

    Query-agnostic streaming video systems must preserve useful information in bounded memory before future queries are known, while coupled memory-design choices are difficult to optimize directly.

    Source: §1 · pp. 1–2

  • Method

    MemEvo searches executable memory programs with an LLM that proposes and refines candidates from experimental feedback, while a deterministic pipeline validates and evaluates them and the MLLM stays frozen.

    Source: §1 · pp. 2–3

  • Results

    Across StreamingBench and OVO-Bench, MemEvo improves frozen backbones; on OVO-Bench average, Qwen3-VL-8B gains 11.5 points over its frozen backbone.

    Source: §1 · p. 3, §4.3 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    The discovered mechanism is a practical alternative to manually engineering one streaming-memory mechanism, with complementary recent, episodic, and gist roles.

    Source: §5 · p. 10

Abstract

from arXiv · show

Query-agnostic streaming video understanding requires vision-language models to continuously compress an indefinitely growing visual stream into a bounded memory before future queries are known. The performance depends critically on the memory mechanism--what observations to preserve, how to represent and consolidate them, and what information to retrieve when a query eventually arrives. Rather than designing a single memory architecture by hand, we formulate memory design as a search problem over executable memory programs. We introduce a lightweight domain-specific language that expresses memory mechanisms through structured primitives for representation, admission, retention, consolidation, budgeting, and retrieval, while enforcing causal and bounded-memory constraints. Although structured, the derived program space remains large and contains heterogeneous, conditionally dependent design choices whose effects can only be assessed via downstream execution. We therefore propose MemEvo, an LLM-driven auto-research framework that uses pretrained LLM as a semantics-aware proposal model to iteratively generate and refine candidate memory programs based on accumulated experimental feedback. At runtime, a deterministic evaluation pipeline validates and evaluates each candidate, while the underlying vision-language model remains frozen throughout discovery. We finally produce a training-free, bounded-memory mechanism. Extensive experiments on StreamingBench and OVO-Bench demonstrate strong streaming video understanding performance together with substantial context and inference efficiency.

1 INTRODUCTION

Query-agnostic streaming video understanding must preserve useful information in bounded memory before future queries are known. MemEvo searches executable memory programs and discovers a mechanism that improves frozen backbones across benchmarks while maintaining bounded memory and reducing context length and inference latency.

1 INTRODUCTION

  • Streaming systems must compress indefinitely growing visual histories into bounded memory before future queries are known, making early discards potentially consequential.Retaining all history could be computationally infeasible.Source: §1 · p. 1
  • MemEvo organizes memory into recent, episodic, and gist roles, then searches coupled choices spanning representation, admission, retention, consolidation, budgeting, and retrieval.Recent memory anchors immediate perception, episodic evidence preserves sparse high-fidelity historical observations, and gist memory summarizes longer ranges.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • MemEvo improves frozen backbones across StreamingBench and OVO-Bench while maintaining bounded memory, with complementary memory roles and reduced reader context length and inference latency.The introduction also reports substantial improvements over frozen backbones, complementary memory components, and favorable context and inference efficiency.Source: §1 · p. 3
  • The paper formulates memory design as a search over executable programs with explicit causal and bounded constraints, rather than manually choosing one architecture.The program specifies how observations update bounded state and how memory is read for a future query.Source: §1 · pp. 2–3
  • MemEvo uses an LLM as a semantics-aware proposal model to iteratively refine candidate memory programs from experimental feedback, while a deterministic pipeline validates and evaluates them.The underlying MLLM remains frozen; experimental outcomes guide which candidates are retained and subsequent exploration.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work addresses long and streaming video memory through bounded banks, compressed or persistent memories, and inference-time retrieval or compression. MemEvo differs by searching the memory mechanism itself and applying LLM-based algorithm discovery to executable memory programs evaluated in downstream experiments.

2 RELATED WORK

  • Long- and streaming-video systems use bounded or compressed memory banks, contextual or event memories, and training-free inference-time retrieval, compression, or retention methods.These approaches establish memory design as important but primarily use manually specified mechanisms.Source: §2 · p. 3
  • Unlike prior manually specified video-memory mechanisms or general algorithm-search systems, MemEvo searches a structured space of memory operations while the underlying MLLM remains frozen.The searched choices include representation, admission, retention, consolidation, budgeting, and retrieval.Source: §2 · p. 3
  • LLM-based discovery systems use language models to generate or modify algorithms and systems under empirical feedback, including EoH, ADAS, and AlphaEvolve.MemEvo applies this paradigm to a domain-specific search over streaming-video memory programs evaluated through downstream experiments.Source: §2 · p. 3

3 METHOD

MemEvo formulates query-agnostic streaming memory design as search over executable programs, using a constrained DSL and LLM proposals refined through experimental feedback. The task requires causal memory updates before queries are known, followed by query-conditioned retrieval and packing.

3 METHOD

  • The method casts memory design as search over executable programs in a compact DSL that fixes high-level structure, interfaces, and causal and resource constraints.Source: §3 · pp. 3–4

3.1 QUERY-AGNOSTIC STREAMING TASK

  • Before a query arrives, programs update bounded state causally, process each frame once, and cannot revisit the video; query-conditioned retrieval follows the cutoff.Source: §3.1 · p. 4

3.2 A DSL FOR MEMORY MECHANISMS

  • The DSL fixes the R/E/G roles, interfaces, frozen models, and packing order, while allowing strategy slots and contract-compliant operators to evolve.Source: §3.2 · p. 4

3.3 LLM-GUIDED PROGRAM SEARCH

  • An LLM proposes finite edits using experimental history and quality-cost trade-offs; staged search freezes components while the evaluation harness remains fixed.Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments discover and evaluate a memory program using sequential, feedback-guided search, then assess its streaming-video performance, memory bounds, and inference efficiency. Results show gains across benchmarks and task types, while memory-role benefits vary by task.

4.1 EXPERIMENTAL SETUP

  • Experiments evaluate Qwen2.5-VL-7B and Qwen3-VL-8B against training-based and training-free baselines at 1 FPS.Search uses NExT-GQA validation with 3,358 questions over 567 videos; the selected program is frozen before evaluation on StreamingBench RTVU and OVO-Bench RT and BT.Source: §4.1 · pp. 6–7

4.2 MEMORY PROGRAM DISCOVERY

  • Across approximately 54 GPU-hours, sequential search improves the development objective by 5–6 points through episodic evidence, retrieval, and gist-memory stages.The search favors interacting design changes over simply increasing memory size, including deeper episodic retrieval, shorter gist segments, and pooled visual representations.Source: §4.2 · pp. 7–8

4.3 MAIN RESULTS ON STREAMING VIDEO BENCHMARKS

  • Qwen3-VL-8B gains 5.7 RTVU points and 11.5 OVO-Bench points, reaching 81.4 and 68.8, the highest overall scores among evaluated methods.For Qwen2.5-VL-7B, MemEvo improves RTVU by 4.2 points and the OVO-Bench average by 10.6 points.Source: Table 1 · p. 7, §4.3 · p. 8
  • Qwen3-VL-8B ranks first in 7 of 10 StreamingBench subcategories and improves over its backbone in eight, with gains of 14.1 on Clips Summarization and 10.8 on Text-Rich Understanding.Causal Reasoning and Counting remain below the unaugmented backbone.Source: Table 2 · p. 8, §4.3 · p. 8
  • Episodic memory helps Object Perception, Event Understanding, and Prospective Reasoning, while gist memory helps Counting, Event Understanding, and Prospective Reasoning.Additional historical evidence can hurt some categories; combining episodic and gist memory recovers part of the loss and reaches the highest score.Source: §4.3 · p. 9
  • 86% lower reader context (nearly 35K to approximately 5K tokens) and 71% lower end-to-end latency versus uniform sampling on OVO-Bench.For StreamingBench prefixes up to 1,024 frames, reader context and stored payload stabilize at 5,500 tokens and 96 MiB.Source: Figure 4 · p. 9, §4.3 · p. 9

5 CONCLUSION

The conclusion presents MemEvo as an LLM-driven framework for discovering executable streaming-memory programs. Its discovered program improves frozen backbones while maintaining bounded memory and efficient inference, and analysis points to interactions among memory-design choices.

5 CONCLUSION

  • MemEvo automatically discovers streaming-video memory mechanisms by searching executable programs rather than manually designing a single architecture.The programs use a compact DSL with causal and bounded-resource constraints, and an LLM iteratively refines proposals using fixed-runtime feedback.Source: §5 · p. 9
  • The discovered program substantially improves frozen backbones while maintaining bounded memory and efficient inference.Experiments cover StreamingBench and OVO-Bench.Source: §5 · p. 10
  • Analysis finds complementary roles for recent, episodic, and gist memory, with effective design depending on interactions among representation, storage, and readout choices.The authors describe automated memory-program discovery as a practical alternative to manually engineering one streaming memory mechanism.Source: §5 · p. 10

AI USE STATEMENT

The authors describe generative-AI assistance in manuscript preparation and research, alongside their review and validation responsibilities. The search section explains how MemEvo evolves programs while preserving a fixed skeleton and resource constraints.

AI USE STATEMENT

  • Generative AI assisted with drafting and revising manuscript text for clarity, organization, and language.The authors reviewed and edited the AI-assisted text.Source: AI USE STATEMENT · p. 10
  • AI tools also assisted with experimental implementation, debugging, and interpretation; GPT-5.6 proposed and refined memory programs and implemented DSL operators using experimental feedback.The authors inspected AI-assisted code and checked analyses against experimental outputs.Source: AI USE STATEMENT · p. 10

A DETAILED EVOLUTION PROCESS OF AUTOMATED RESEARCH

  • The search modifies exposed DSL operators and parameters while preserving a fixed program skeleton, causal interfaces, and resource constraints.It does not optimize stored contents for individual videos.Source: A DETAILED EVOLUTION PROCESS OF… · p. 14
  • The search evaluated 89 canonically distinct programs across 13 refinement rounds; 85 completed successfully and four failed.The stages followed the schedule E → Retrieval → G, with candidates deduplicated before evaluation.Source: A DETAILED EVOLUTION PROCESS OF… · p. 14
  • Search rounds expose different program semantics, and later rounds can recombine useful choices discovered in separate branches.Edits include admission, retention, segmentation, representation, and retrieval.Source: A DETAILED EVOLUTION PROCESS OF… · p. 14

A.1 EVOLUTION OF EPISODIC MEMORY

The episodic-memory search varied capacity, admission, temporal constraints, and retention before selecting a retrieval depth that favored episodic evidence over gist. The selected retrieval configuration achieved the stage-best validation score, while greater gist depth did not improve it.

A.1 EVOLUTION OF EPISODIC MEMORY

  • 0.735083 was reached by capacity-16 reservoir retention, while the three capacity-8 variants scored approximately 0.686–0.689.The round tested capacities from 8 to 64 entries with FIFO, temporal, and reservoir-style retention.Source: §A.1 · p. 14
  • 0.730817 was reached by optical-flow-q75 admission, compared with 0.697551 for histogram-q90 and 0.695995 for optical-flow-q90.The search retained histogram-q50 for further refinement rather than simply freezing the locally strongest trigger.Source: §A.1 · p. 15
  • 0.739975 was the running best after adding a minimum temporal gap of two to histogram-q50 admission; maximum-gap variants scored around 0.7527–0.7534.Minimum gaps suppress temporally redundant evidence, whereas maximum gaps ensure an observation is admitted after sufficiently long intervals.Source: §A.1 · p. 15
  • 0.753435 was obtained by all nine coverage-aware retention variants, with optical-flow coverage at weight 0.75 selected as the episodic-stage winner.The validation scores did not distinguish their retention behavior; selection followed the designated episodic-stage procedure.Source: §A.1 · p. 15
  • 0.740043 and 0.740008 were obtained by histogram-q50 admission with minimum-gap-2 under two maximum-gap settings; reducing coverage-based capacity to 32 scored 0.738682.The selected E4 episodic program was then held fixed in subsequent stages.Source: §A.1 · p. 15

A.2 EVOLUTION OF QUERY-TIME RETRIEVAL

  • 0.771990 was the stage-best score at retrieval depths (kE, kG) = (8, 4), while increasing gist retrieval to kG = 8 yielded 0.771109.The selected program retrieves substantially more episodic evidence than gist entries; increasing gist depth from four to eight slightly reduced validation score.Source: §A.2 · p. 15

A.3 EVOLUTION OF GIST MEMORY

Gist-memory search explored segmentation, encoding, capacity, and consolidation, then tested a hybrid visual–textual representation. The strongest reported hybrid candidate scored 0.790721, while adaptive segmentation and alternative consolidation did not improve the simpler fixed-segment branch.

A.3 EVOLUTION OF GIST MEMORY

  • The gist DSL search covered temporal segmentation, textual representation, global capacity, consolidation, and hybrid visual–textual encoding.Its core program form is Segment, then Encode, then Merge under a bounded gist budget.Source: §A.3 · p. 15
  • 0.784083 was reached by fixed 8-frame segments with text budgets of either 128 or 256 tokens, while fixed 32-frame segments scored 0.776876.The two fixed-8 variants were equal on this validation set, and fixed(8)+text(128) was retained as a final Pareto candidate.Source: §A.3 · p. 16
  • 0.786595 was reached with fixed 8-frame segments, 128 text tokens per entry, and a global text capacity of 512.Global capacities of 1024 and 4096 preserved the prior 0.784083 score; the tighter budget became a running best and final Pareto candidate.Source: §A.3 · p. 16
  • 0.778048 followed most-similar-entry consolidation, while removing consolidation failed; neither candidate improved the current anchor.The failed execution remained in search history as feasibility feedback, but not in the empirical quality–cost archive.Source: §A.3 · p. 16
  • 0.779277 was the best adaptive-boundary refinement, below the fixed-segment anchor at 0.786595; a one-frame minimum scored 0.743346.Optical-flow-q50 reached 0.777711, and the search returned to fixed segmentation for later representation exploration.Source: §A.3 · p. 16
  • 0.790721 was reached by a hybrid gist using 1/4 pooling, a 128-token per-entry visual cap, and a global visual budget of 1024.Increasing the global cap to 2048 preserved the score, while denser representations or a 256-token per-entry cap performed worse.Source: §A.3 · p. 16

A.4 RESULTING MEMORY PROGRAM

The resulting program combines choices discovered across the three search stages. MemEvo’s search evaluates coupled edits empirically and removes its research machinery before serving-time inference.

A.4 RESULTING MEMORY PROGRAM

  • The final memory program combines decisions discovered across the three stages.The paper points to Figure 2 for the resulting combination.Source: §A.4 · p. 17
  • Useful improvements arise from coupled program edits, and more sophisticated operators are retained only when execution supports them.The search preserves and revisits multiple semantic branches, recombines components, and uses failed executions as feasibility feedback.Source: §A.4 · p. 17
  • At serving time, the research agent is removed; only the frozen program, bounded online state, query-time readout, and frozen MLLM remain.Source: §A.4 · p. 17

B DETAILED SETTINGS OF BASELINES

The baseline settings section reports that its experimental results are collected from published papers and directs readers to Table 4 for method details. The table summarizes compared methods’ base models, publication information, and data sources.

B DETAILED SETTINGS OF BASELINES

  • Section 4 experimental results are collected from published papers.Source: B DETAILED SETTINGS OF BASELINES · p. 17
  • Table 4 summarizes the base models, publication information, and data sources for compared methods.Source: Table 4 · p. 17
  • “Official” in Table 4 means that scores come from the method’s own paper; otherwise, the source is stated explicitly.Source: Table 4 · p. 17

C RESOURCE PROFILING OF MEMEVO

Profiling reports stable reader memory size and runtime, and a pipelined writer operating at 1 FPS. Edge deployment would still require hardware-specific optimization.

D CASE STUDY

The cases show memory roles supporting a correct answer, but also show historical context confusing temporal order. The prompt uses timestamps and avoids duplicate visual evidence.

  • When recent frames overlap retrieved episodic evidence, the pipeline checks for duplicates and avoids presenting the same visual evidence twice, reducing context use and computation.Source: D CASE STUDY · pp. 18–19
  • The timestamp-aware prompt marks query and memory times, labels gist spans with start and end times, and instructs how to interpret and integrate the three memory roles.Source: D CASE STUDY · p. 18, Figure 5 · p. 19
  • At 7.0 s, recent frames, top-k episodic retrieval, and a short-video gist provide complementary cues for the correct kitchen-table answer.Source: D CASE STUDY · pp. 19–20, Figure 6 · p. 19
  • At 432.4 s, retrieved episodic evidence and gist surface earlier events despite recent memory indicating rabbit release, confusing event order and yielding a wrong answer.Source: Figure 7 · p. 20, D CASE STUDY · p. 20
  • Current retrieval does not explicitly account for temporal relationships, while single-pass logits leave autoregressive test-time reasoning and its context–performance trade-off for future work.Source: D CASE STUDY · p. 20
Loading 2609.36581v1…