PaperlayerView exact revision on arXiv

Source-linked AI summary

KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding

Zihan Chen, Xuejian Rong, Xiaojuan Wang, Boqing Gong, Adi Zicher, Yael Pritch, Nikhil Karnad

arXiv:2609.32182v1cs.CVcs.AI

TL;DR

Dense visual tokens make streaming and long-video inference expensive, and existing token-selection methods can lose coherent event evidence or fine-grained recent context. KeyRec combines recent and event memory with adaptive fixed-budget readout; across four benchmarks and three VLM backbones, it achieves the best compressed performance in 13 of 15 settings using 10% of the dense visual-token budget.

  • Problem

    Dense visual sequences grow with video duration, making long-context inference costly for streams with arriving queries and for long videos with limited memory or repeated questions.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    KeyRec builds a dense recent cache and bounded event memory, then uses a text-only router for query-adaptive readout without reprocessing historical frames.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    At 10% of the dense decoder-facing visual-token budget, KeyRec achieves the best compressed performance in 13 of 15 settings across four benchmarks and three VLM backbones.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments indicate that organizing and presenting visual evidence matters alongside selecting informative tokens, particularly for questions requiring fine-grained recent context.

    Source: §4 · p. 10

    Existing token-selection approaches may fragment evolving events, conflate visually similar occurrences at different times, or discard context linking actors, actions, objects, and scenes.

    Source: §1 · pp. 1–2

Abstract

from arXiv · show

Vision-language models are increasingly used to understand long videos and continuous streams. However, dense visual tokens accumulate with video duration, making long-context inference prohibitively expensive. Existing training-free visual-token selection methods reduce this cost by retaining informative tokens, but may lose coherent event evidence and fail to distinguish detailed recent observations from long-range history. We propose \textbf{KeyRec}, a training-free framework for constructing bounded visual memory. During query-agnostic writing, KeyRec preserves fine-grained recent observations in a visual cache and organizes historical evidence into a structured event bank. Candidate events are proposed according to their novelty relative to previously stored events and maintained through an online add--merge--evict update. When a question arrives, a text-only router adaptively allocates a fixed readout budget between recent and event memory, without reprocessing historical frames. KeyRec operates on model-facing visual embeddings and supports both modular encoder--projector VLMs and the encoder- and projector-free NEO-ov architecture. Across four streaming and long-video benchmarks and three VLM backbones, KeyRec achieves the best compressed performance in 13 of 15 settings using only 10\% of the dense decoder-facing visual-token budget. It outperforms the strongest compressed baseline by 2.21--18.37 points on real-time questions, achieves the best compressed result in five of six long-video settings, and performs best in every NEO-ov 2B setting.

1. Introduction

Dense visual tokens make long-video and streaming inference costly, while existing compression methods can lose coherent event evidence and detailed recent context. KeyRec builds bounded recent and historical memory and adaptively reads it, achieving the best compressed performance in 13 of 15 settings at 10% of the dense visual-token budget.

1. Introduction

  • Because frame-level visual tokens grow linearly with video duration and dense self-attention costs grow quadratically with sequence length, long videos and streams strain context and computation budgets.Streaming queries may arrive at any time, while long-video questions may repeatedly process historical frames.Source: §1 · p. 1
  • Earlier training-free methods reduce redundant visual context, but may fragment evolving events, conflate distinct occurrences, or discard context linking actors, actions, objects, and scenes.A common retention policy can also preserve broad semantics while losing fine-grained evidence for questions about what is happening now.Source: §1 · pp. 1–2
  • KeyRec combines a dense recent cache with structured historical events maintained by an online add–merge–evict update and a query-adaptive, fixed-budget readout.The text-only router allocates readout between recent and event memory without reprocessing historical frames.Source: §1 · p. 2
  • At 10% of the dense decoder-facing visual-token budget, KeyRec achieves the best compressed result in 13 of 15 settings across four benchmarks and three VLM backbones.The framework is training-free.Source: §1 · p. 2

2. Method

KeyRec formalizes bounded visual memory as query-agnostic writing followed by query-adaptive, fixed-budget reading. It separates detailed recent evidence from historical event memory and adapts the method to NEO-ov with positional bookkeeping.

2.1. Problem Formulation and Overview

  • KeyRec stores recent and historical visual evidence in bounded memory, then reads it under a fixed per-query budget without revisiting the video.The recent cache preserves latest-frame evidence, while the event bank retains historical evidence.Source: §2.1 · pp. 2–3

2.2. Query-Agnostic Memory Construction

  • The recent cache retains the latest N_C tokens, while a bounded key-event memory stores at most K events with up to m visual tokens each.Together, these stores fit within the persistent storage budget S.Source: §2.1 · p. 2, §2.2 · p. 3
  • Each frame proposes an event using tokens novel relative to stored event routes; candidates are added, merged, or evicted, with temporal adjacency constraining merges.Consolidation also considers semantic similarity and temporal proximity; event tokens are time-sorted and uniformly subsampled when they exceed per-event capacity.Source: §2.2 · pp. 3–4

2.3. Query-Adaptive Fixed-Budget Readout

  • A text-only router selects among five recent-to-event readout states, allocating the fixed budget and choosing historical events by relevance or temporal coverage.HeavyRecent and Balanced use route similarity to the question; HeavyEvent and AllEvent select events across the sequence by time.Source: §2.3 · pp. 4–5

2.4. KeyRec on Native One-Vision Models

  • KeyRec applies to NEO-ov using model-facing visual embeddings, preserving each retained token’s positional metadata and restoring its native positions at read time.The memory algorithm is unchanged; the adaptation requires positional bookkeeping rather than dense visual-grid reconstruction.Source: §2.4 · p. 5

3. Experiments

The experiments compare KeyRec with frozen-backbone compression baselines across streaming and long-video benchmarks, then probe readout scaling, frame scaling, routing, efficiency, and full-prefix streaming. KeyRec achieves the best compressed result in most main benchmark settings, with gains also reported under fixed-budget full-prefix evaluation.

3.1. Experimental Setup

  • The setup covers streaming and long-video benchmarks, three frozen VLM backbones, and compressed methods evaluated at approximately 10% of the dense decoder-facing budget.Streaming uses sampled prefixes; the main comparison caps each prefix at 32 frames, with fixed-budget full-prefix 1-FPS streaming evaluated separately.Source: §3.1 · pp. 5–6

3.2. Main Experimental Results

  • KeyRec achieves the best compressed performance in eight of nine streaming settings, five of six long-video settings, and every NEO-ov 2B setting.At a matched approximately 10% decoder-facing budget, it matches or exceeds Vanilla in all nine streaming settings and outperforms the strongest compressed baseline by 2.21–18.37 points on OVO-Bench Realtime and StreamingBench.Source: Table 1 · p. 6, §3.2 · pp. 6–7

3.3. Scaling with the Decoder-Facing Readout Ratio

  • At matched persistent storage, KeyRec achieves higher accuracy while exposing half as many decoder-facing visual tokens as the baselines.Across both backbones, it peaks at a 5% readout and then gradually declines as visual context increases; peak configurations outperform uncompressed Vanilla.Source: Figure 2 · p. 7, §3.3 · p. 7

3.4. Scaling with the Number of Sampled Frames

  • On 900–3,600-second videos, KeyRec performs best from 64 frames onward on Gemma E4B and remains stable across 32–512 frames on NEO-ov.At 512 frames on NEO-ov, KeyRec stays above 44% accuracy while STC-Pruner and CausalMem are below 25%; where feasible, compressed methods generally remain below Vanilla.Source: §3.4 · pp. 7–8, Figure 3 · p. 8

3.5. Effect of Query-Adaptive Memory Allocation

  • With the same constructed memory and decoder-facing budget, the router achieves the best or tied performance across settings by adapting recent–event allocation to each question.Backward reasoning favors event memory, real-time questions favor recent evidence, and LongVideoBench performs best with a mixture.Source: Figure 4 · p. 8, §3.5 · p. 8

3.6. Latency and Memory Analysis

  • Relative to STOM-CTR, KeyRec’s end-to-end overhead falls from approximately 19% at 64 frames to 10% at 512 frames while remaining feasible at 512 frames.Vanilla runs out of memory at 512 frames, and KeyRec uses moderately more memory than STOM-CTR due to its larger candidate memory.Source: Table 2 · p. 9, §3.6 · p. 9

3.7. Fixed-Budget 1-FPS Streaming Evaluation

  • Under fixed-budget full-prefix 1-FPS streaming, KeyRec achieves higher overall performance than CausalMem on both OVO-Bench and StreamingBench across two backbones.The decoder-facing budget is independent of prefix length; KeyRec’s advantage persists without the 32-frame cap, with persistent storage and per-query decoding cost bounded independently of stream duration.Source: Table 3 · p. 9, §3.7 · p. 9

4. Conclusion

KeyRec builds bounded visual memory from structured historical events and detailed recent evidence, then adapts readout to the question. Experiments emphasize evidence organization and the challenge of adapting compression to encoder-free VLMs.

4. Conclusion

  • KeyRec combines structured historical events, detailed recent evidence, and query-adaptive readout in a training-free bounded-memory framework.The paper concludes that organizing and presenting evidence matters as much as selecting individually informative tokens, particularly for questions requiring fine-grained recent context.Source: §4 · p. 10
  • Across streaming and long-video benchmarks, evidence organization matters as much as selecting informative tokens, especially for fine-grained recent-context questions.The conclusion frames this as an experimental finding about KeyRec, not a universal prescription.Source: §4 · p. 10
  • Adapting visual memory to native representations of encoder-free VLMs remains a consequential direction for video understanding.The conclusion notes that architecture-specific signals, positional mechanisms, and visual-token interfaces can change how transferred compression methods perform.Source: §4 · p. 10

Default

The Default serialization preserves memory boundaries with neutral segment labels; the accompanying template depicts selected visual memory segments and video markers.

Default

  • Selected visual memory is represented as separate segments, each paired with its token count and visual-token embeddings.The illustration labels two segments with token counts n1 and n2.Source: Default · p. 14
  • The video input illustration consists of repeated <|video|> markers.Source: Default · p. 14
  • Default preserves memory boundaries using neutral segment labels, unlike Naive boundary removal and Detailed labels exposing memory type and temporal range.These are the three memory-serialization templates described in the figure caption.Source: Figure 5 · p. 14

A.1. Effect of Memory Serialization

The serialization comparison favors preserving memory-block structure with neutral labels, while event-granularity results favor intermediate-sized evidence units under a fixed storage budget.

A.1. Effect of Memory Serialization

  • With identical embeddings, order, and readout budget, Default keeps separate blocks and neutral labels; Naive flattens them, while Detailed adds type and temporal descriptions.The authors suggest semantic labels may bias evidence weighting, whereas neutral labels avoid those extra cues.Source: §A.1 · p. 14
  • Default generally achieves the best performance across evaluated settings, supporting preserved memory organization during presentation to the language model.Table 4 compares variants using identical selected visual evidence and different textual scaffolds.Source: Table 4 · p. 14, §A.1 · p. 14

A.2. Sensitivity to Event Granularity

  • The event-granularity comparison varies per-event token size and event-bank capacity while holding total event capacity and recent/event readout allocation fixed.This isolates how the fixed event-memory budget is organized rather than changing the number of tokens stored or presented.Source: §A.2 · pp. 14–15
  • m=16 performs best on both benchmarks under fixed event capacity and readout allocation.At m=8 each event may be incomplete; at m=32 fewer distinct events fit, particularly harming OVO-Bench backward reasoning.Source: Table 5 · p. 15, §A.2 · p. 15

A.3. Robustness to Consolidation Thresholds

On OVO-Bench Backward, KeyRec remains stable around its default consolidation thresholds, though a strict similarity criterion paired with a wide temporal window performs worse.

A.3. Robustness to Consolidation Thresholds

  • The sweep varies route-similarity threshold γ across 0.65, 0.75, and 0.85 and maximum temporal gap δ across 1, 2, and 4 seconds.All other settings remain fixed.Source: §A.3 · p. 15
  • The default threshold setting is within 0.22 points of the best OVO-Bench Backward result, with nearby combinations obtaining comparable accuracy.The table caption identifies the evaluation as OVO-Bench Backward with Gemma 4 E4B; the default is marked with †.Source: Table 6 · p. 15, §A.3 · p. 15
  • Performance falls more noticeably when strict similarity is combined with a wide temporal window, a configuration that may obscure recurring-event distinctions.Source: §A.3 · p. 15

A.4. Router Behavior Across Backbones

Across backbones, realtime questions favor recent memory, while backward questions shift toward balanced or event-oriented memory. The routing granularity varies by model, but the task-dependent memory orientation is consistent.

A.4. Router Behavior Across Backbones

  • Gemma 4 E2B favors AllRecent and Balanced, Gemma 4 E4B more often selects HeavyRecent, and NEO-2B assigns more backward questions to HeavyEvent.The fine-grained routing distributions are model-dependent.Source: §A.4 · p. 15
  • Realtime questions favor recent-oriented routing, while backward questions shift toward balanced or event-oriented states across backbones.Figure 6 shows the five routing-state distributions on OVO-Bench.Source: §A.4 · p. 15, Figure 6 · p. 16
  • The consistent shift toward recent evidence for realtime understanding and historical evidence for backward reasoning supports query-adaptive allocation over a single recent–event split.Source: §A.4 · p. 16

B. Additional Implementation Details

KeyRec uses shared operating profiles to bound memory and readout across benchmarks, with separate settings for streaming and long-video evaluation. A StreamingBench case study illustrates how event retrieval covers historical evidence under a fixed budget.

B. Additional Implementation Details

  • OVO-Bench and StreamingBench share the streaming profile, while LongVideoBench and Video-MME-v2 share the long-video profile, with no benchmark-specific tuning.Source: B. Additional Implementation Details · p. 16
  • The streaming and long-video profiles process at most 32 and 64 frames, respectively, and expose a decoder-facing budget equal to 10% of dense visual tokens.For Gemma streaming, the cache and event bank store 224 and 256 tokens, respectively, while only 10% of the dense input is exposed per query.Source: B. Additional Implementation Details · p. 16
  • Streaming event candidates may be merged when semantic and temporal criteria are met, whereas long-video retention uses visual utility across the sampled complete horizon.Streaming also enables temporal coverage to avoid concentrating the bounded event bank on a narrow portion of an unknown horizon.Source: B. Additional Implementation Details · p. 16
  • Complete-event retrieval discretizes the readout into 14 event blocks for streaming and 28 for long video, using a shared codebook across backbones within each profile.Source: B. Additional Implementation Details · p. 16, Table 8 · p. 17
  • Compressed baselines use the same frozen backbone, sampled frames, prompts, and answer extraction as KeyRec, and all use a 10% decoder-facing retention ratio.StreamingTOM-CTR is evaluated only on Gemma because it requires signals from a dedicated vision encoder.Source: B. Additional Implementation Details · pp. 16–17

C. Qualitative Case Study

  • In the NEO-ov 2B case study, the router selects history coverage for a question about why thieves enter undetected, and the readout combines events from both portions of the history with recent evidence.Visually similar event pairs remain separate when their temporal gaps exceed the merge threshold; the selected events support the inference that the masters have fallen asleep.Source: C. Qualitative Case Study · p. 17

D. Per-Task and Category-Level Results

Per-task and category-level results show that KeyRec's gains vary across benchmarks, tasks, and backbones.

D. Per-Task and Category-Level Results

  • KeyRec has the best overall StreamingBench accuracy on all three backbones, while OVO-Bench gains are most consistent on real-time tasks and backward-tracing results are more mixed.Some StreamingBench categories, such as Cnt, remain challenging.Source: D. Per-Task and Category-Level Results · p. 17
  • On Video-MME-v2, KeyRec is the strongest compressed method across reported rating dimensions for Gemma 4 E4B and NEO-ov 2B, while Gemma 4 E2B results are more competitive across methods.Source: D. Per-Task and Category-Level Results · p. 17

E. Limitations

The limitations include fixed memory policies, no reduction in Gemma ViT encoding latency, and limited explicit representation of object-level temporal dynamics.

E. Limitations

  • KeyRec uses fixed novelty, consolidation, and retention rules, which may be improved by learning from token-level supervision or downstream QA signals.Source: E. Limitations · p. 17
  • In the Gemma implementation, post-encoder memory reduces visual-memory and language-model costs but not ViT encoding latency; efficient encoder-side integration remains an open systems challenge.Source: E. Limitations · p. 17
  • Selected visual patches do not explicitly model object identities, trajectories, or state transitions, limiting precise reasoning about duration, order, and cross-frame evolution.Source: E. Limitations · p. 17

Related papers

Loading 2609.32182v1…