PaperlayerView exact revision on arXiv

Source-linked AI summary

ReMem: Streaming Video Understanding With Long Context Retention

Li Yiheng, He Xu, Wang Shaobo, Shao Ling, Lu Shijian

arXiv:2610.05940v1cs.CVcs.AI

TL;DR

Offline VLMs are not designed for continuous video streams that require low-latency responses, and existing streaming methods can lose long-context information as streams grow. ReMem is a training-free adaptation with complementary memory components for compression and retrieval. The paper reports that it outperforms baselines and competing memory-based methods across online and general long-video benchmarks.

  • Problem

    Offline VLMs presume the full visual input is available before response generation, while streaming methods can progressively lose long-context information as input length increases.

    Source: Introduction · p. 1

  • Method

    ReMem is a training-free adaptation that uses query-independent Streaming Context Memory compression and Retrieved Vision Memory to add query-relevant visual information.

    Source: Introduction · p. 1, Conclusion · p. 7

  • Results

    ReMem significantly outperforms baseline models and competing memory-based methods on online and general long-video benchmarks, and its NIAH probing confirms long-term context preservation.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    ReMem is presented as a flexible, training-free way to deploy pretrained VLMs in real-world applications while retaining long-term context.

    Source: Introduction · p. 2, Conclusion · p. 7

    Query-based compression can discard information irrelevant to the current prompt but needed for subsequent queries in multi-turn streaming.

    Source: Visual Token Compression · p. 2

Abstract

from arXiv · show

Despite their impressive performance on a wide range of video understanding tasks, current Vision Language Models (VLMs) are predominantly designed for offline scenarios and struggle to handle online streaming videos that demand low latency response. Several studies have explored memory and token compression strategies in an attempt to adapt offline VLMs for streaming video understanding tasks. However, through our probing experiment, we identify that most existing works tend to progressively lose long context information as length of input stream increases. To address this, we propose ReMem, a novel training-free adaptation technique that enables VLMs to process streaming videos of arbitrary lengths while improving their long context information retention capability. ReMem exploits memory from two perspectives, implemented as two core components. The Streaming Context Memory (SCM) continuously compresses historical context with query-independent attention. The Retrieved Vision Memory (RVM) then retrieves the most salient, query-relevant context from memory to augment the VLM's input. Comprehensive experiments demonstrate that the proposed ReMem achieves state-of-the-art (SOTA) performance across a variety of widely used benchmarks, spanning both streaming video and general long video understanding tasks.

Abstract

Offline VLMs struggle to meet the low-latency demands of continuous video streams, and existing streaming approaches can lose long-context information as inputs grow. ReMem is a training-free adaptation that combines two memory components and is reported to outperform baselines across streaming and long-video benchmarks.

Abstract

  • Offline VLMs presume access to the complete visual input before responding, an assumption that may not suit real-time applications requiring low latency.The paper gives autonomous driving as an example of an application expecting fast responses.Source: Introduction · p. 1
  • Streaming Context Memory continuously compresses historical visual information, while Retrieved Vision Memory retrieves query-relevant vision tokens to augment the model input.SCM uses attention-based compression; RVM retrieves the most relevant, query-specific tokens.Source: Introduction · p. 1
  • Visual NIAH probing indicates that existing memory-based streaming VLMs progressively lose long-term context as the input stream grows.The paper reports that FluxMem and OASIS show a more pronounced retrieval-accuracy drop than the Qwen2.5VL base model as haystack size increases.Source: Introduction · pp. 1–2
  • ReMem adapts offline-pretrained VLMs to streaming video of arbitrary length while improving long-context retention, without additional training.It is designed to retain long-term context while keeping input token length short for efficiency.Source: Introduction · pp. 1–2
  • ReMem significantly outperforms baselines and competing streaming methods across streaming and general long-video benchmarks, particularly on streaming tasks.The experiments are described as comprehensive, with gains especially on tasks requiring contextual understanding.Source: Introduction · pp. 1–2

Related Works

Prior work adapts VLMs for online video through changes to model use, token compression, and memory, but the passages describe constraints around context windows, multi-turn retention, and memory growth. ReMem’s related-work framing emphasizes attention-based compression and retrieval as ways to address these concerns.

Vision Language Models

  • VLMs typically concatenate visual features with text embeddings for autoregressive generation, but their offline design and fixed context windows constrain continuous low-latency streaming use.The described architecture includes a vision encoder, connector, and LLM backbone.Source: Vision Language Models · p. 2

Streaming Video Understanding

Visual Token Compression

  • Query-based compression such as Q-former (Li et al. 2023) can discard information needed for later turns, while spatial-temporal redundancy reduction may require fine-tuning after shifting the input distribution.Attention-based selection instead retains tokens or KV-cache vectors with high attention scores to limit out-of-distribution behavior and avoid additional training.Source: Visual Token Compression · p. 2

Memory Based VLM

  • ReKV (Di et al. 2025b) and MovieChat (Song et al. 2025) retain earlier video information, but at least one module in common designs may lack an upper memory bound.For unbounded streams, accumulated token length can eventually exceed hardware capacity.Source: Memory Based VLM · pp. 2–3
  • Fixed-footprint methods such as InfiniPlotV (Kim et al. 2025) and StreamBridge (Wang et al. 2025a) compress or merge earlier context, whereas the paper describes retrieval as extracting salient information without compressing critical details.The retrieved features augment fixed-size memory to mitigate information loss from progressive memory updates.Source: Memory Based VLM · p. 3

Probing Experiment

The probing experiment tests whether streaming memory systems retain a visual detail placed at the beginning of a sequence as irrelevant images accumulate. ReMem is reported to retain retrieval performance more stably than the evaluated alternatives as the haystack grows.

Probing Experiment

  • The probing tests the hypothesis that fixed-size context memory progressively dilutes or overwrites earlier information as new frames arrive.The paper describes fixed-size memory as memory-efficient and low-latency, but hypothesizes a cost to long-context retention.Source: Probing Experiment · p. 3
  • The Visual NIAH task places a needle image at the sequence beginning, then appends irrelevant haystack images while the model answers a question about the needle.The experiment uses the Visual NIAH dataset and increases context length by adding images.Source: Probing Experiment · p. 3
  • As haystack size increases, ReMem maintains more stable retrieval performance than evaluated models, while FluxMem and OASIS approach 0.50 on the yes/no task.The figure compares Qwen2.5VL, FluxMem, HERMES, OASIS, and ReMem.Source: Figure 1 · p. 3, Probing Experiment · p. 3

Methodology

ReMem adapts offline pretrained VLMs to streaming video using fixed-length context compression and query-relevant visual retrieval. Its components process incoming clips continuously and use attention-based selection to preserve and retrieve visual context.

Methodology

  • ReMem combines a fixed-length Streaming Context Memory (SCM) with a Retrieved Vision Memory (RVM) that reinjects query-relevant visual tokens into the model input.SCM compresses incoming visual information; RVM retrieves and replays relevant context.Source: Methodology · p. 3

Preliminary

  • At each discrete time step, a new C-frame clip is encoded into a segment of vision tokens, Vi.The number of frames per clip can be controlled through the video sampling frequency.Source: Preliminary · p. 3

Streaming Context Memory

  • SCM iteratively compresses prior memory and current vision tokens into a fixed-length state, with its maximum size controlled by compression ratio r.The SCM updates continuously during streaming, including while the VLM generates a response to a user query.Source: Streaming Context Memory · pp. 3–4

General Query Compression

  • GQC uses text-vision attention from a static general-purpose prompt to select tokens independently of the user's query, retaining the highest-scoring tokens under ratio r.Selected tokens retain their original positional embeddings, while a cumulative frame-count offset tracks their absolute temporal positions.Source: Streaming Context Memory · p. 4, General Query Compression · p. 4

Retrieved Vision Memory

  • RVM represents each video clip with an average-pooled key vector and retrieves the clip whose key has the highest cosine similarity to the pooled query vector.Retrieved clips are appended to the current input; the RVM buffer discards its oldest entries when its maximum length is exceeded.Source: Retrieved Vision Memory · p. 5

Experiments

ReMem is evaluated on streaming and offline video benchmarks, with probing and ablations examining context retention and design choices. The reported results show strong streaming performance, preserved long context, and bounded latency and memory use.

Experiment Setup

  • The evaluation applies ReMem to LLaVA-OneVision-7B and Qwen2.5-VL-7B on streaming and offline benchmarks, with comparisons conducted under identical single-H100 hardware and decoding protocols.Streaming evaluation focuses on contextual tasks in StreamingBench and OVOBench; offline evaluation uses EgoSchema and VideoMME.Source: Experiment Setup · p. 5

Efficiency Analysis

  • ReMem outperforms its baseline on EgoSchema and VideoMME, including a 2.4-point gain over LLaVA-OneVision-7B, while remaining comparable on longer VideoMME settings.The reported EgoSchema results use the subset because full-set evaluation was unavailable due to an evaluation-server issue.Source: Main Results · p. 5, Table 2 · p. 7

Probing Experiments Analysis

  • In visual NIAH probing, ReMem achieves consistently higher accuracy than Qwen2.5-VL-7B across nearly all input image-size settings, suggesting it preserves long-term context on par with or slightly better than the strong non-streaming baseline.Source: Probing Experiments Analysis · p. 5

Main Results

  • ReMem outperforms offline baselines and streaming-oriented models on StreamingBench contextual tasks and OVOBench backward tracing, including gains of 20.0 points on ACU and 11.6 on MCU over Qwen2.5-VL-7B.The experiments also report improvements across Qwen2.5-VL-7B, Qwen3-VL-8B, and LLaVA-OneVision-7B base models.Source: Main Results · p. 5, Table 1 · p. 6

Method Ablation

  • With all proposed strategies, the ablation achieves the highest performance gain; SCM, RVM, and position-embedding management each contribute positively over the Qwen2.5-VL-7B baseline.The ablations evaluate SQA and ACU and compare continuous position indices after compression with the proposed position-embedding strategy.Source: Method Ablation · p. 6, Table 3 · p. 6, Further Experiments · p. 6

Hyperparameter Study

  • SCM accuracy on SQA and ACU is lowest at chunk size 1 and peaks at 16, with performance improving as chunk size increases.The study varies chunk size using SCM alone, without RVM retrieval.Source: Hyperparameter Study · p. 6, Table 4 · p. 7
  • Retrieving a single historical clip yields the best StreamingBench performance; the authors suggest additional context may add distracting noise or exceed current models’ memory-perception trade-off.The comparison varies the number of query-relevant historical clips retrieved by RVM.Source: Hyperparameter Study · p. 6, Table 5 · p. 7

Number of Clips SQA MCU ACU

  • On StreamingBench SQA, ReMem maintains 1.0-second latency through 512 input frames and its memory footprint plateaus at 32 GB once the buffer is full.The measurements use compression ratio 0.7, chunk size 16, 1 fps, and a single 80GB H100 GPU.Source: Figure 3 · p. 7, Number of Clips SQA MCU ACU · p. 7

Conclusion

The conclusion presents ReMem as a training-free streaming adaptation designed to address long-context loss in fixed-size-memory VLMs. Its experiments report gains across online and long-video benchmarks and preservation of long-term context in probing.

Conclusion

  • ReMem combines query-independent continuous visual-token compression in Streaming Context Memory with query-specific salient-vision augmentation in Retrieved Vision Memory.The architecture is presented as a response to progressive long-term context loss in fixed-size-memory streaming VLMs.Source: Conclusion · p. 7
  • Experiments on online and general long-video benchmarks report that ReMem significantly outperforms baseline models and competing memory-based methods.NIAH probing also reports preserved long-term context without the pathological information loss observed in other streaming approaches.Source: Conclusion · p. 7
  • As a flexible, training-free method, ReMem is presented as enabling deployment of pretrained VLMs in real-world applications.Source: Conclusion · p. 7

Additional Experimental Details

The section adds experimental analyses and reports ReMem evaluations beyond the primary contextual-reasoning focus, including broader streaming-benchmark settings. It also examines prompting strategies and compression choices.

Full Streaming Benchmark Results

  • The broader evaluation reports StreamingBench real-time visual understanding and OVOBench real-time perception results alongside the primary contextual-reasoning tasks.Table 6 covers ten StreamingBench tasks; Table 7 covers six real-time and three backward-tracing OVOBench tasks.Source: Full Streaming Benchmark Results · p. 10, Table 6 · p. 10, Table 7 · p. 11

Prompting Strategies Analysis

  • 92.7 average cosine similarity is achieved by the general prompt, versus 85.7 for the chat-template prompt and 82.1 for hooking.The general prompt asks for a sequential description from start to finish; experiments use 250 StreamingBench SQA samples, compression ratio 0.7, chunk size 16, and 1 fps.Source: Prompting Strategies Analysis · pp. 10–11, Table 9 · p. 11

Compression Experiment

  • At r = 0.7, performance is nearly lossless for GQC and VisionZip, while performance degrades as compression increases.Performance is lowest for both methods at r = 0.1; the authors select r = 0.7 as their accuracy-compression balance for subsequent experiments.Source: Table 8 · p. 11, Compression Experiment · p. 12
  • The general-query and task-query attention distributions remain spatially consistent over visual tokens despite differing absolute scores.Figure 4 illustrates attention-score distributions for a task query and a general sequential-description query.Source: Compression Experiment · p. 12, Figure 4 · p. 12

Method Elaboration

The General Query Compression algorithm selects visual tokens using attention scores, reducing the input to a chosen fraction of tokens.

Method Elaboration

  • The algorithm takes vision tokens, attention weights, image-token boundaries, and a compression ratio as inputs.The pseudocode identifies these as Vi, Attn, Istart, Iend, and r.Source: Method Elaboration · p. 12
  • It averages attention across layers and heads, then extracts image-token attention and averages those scores across text queries.The resulting score vector assigns an aggregate attention score to each visual token.Source: Method Elaboration · p. 12
  • It selects the top floor(r × Nimg) visual tokens by aggregate score and returns those tokens.Nimg is the total number of vision tokens; the selected indices are the top-scoring entries in Simg.Source: Method Elaboration · p. 12

Case Studies

The case studies compare ReMem-enhanced and Qwen2.5-VL baseline answers on misleading-context, anomaly-context, and sequential-question examples. ReMem answers the cited examples correctly where the baseline does not.

Case Studies

  • In three misleading-context examples, ReMem gives the ground-truth answer while the baseline is distracted by earlier frames or evolving object states.The examples concern Bridge symbols, a scorecard behind a curtain, and face-up cards at a casino table.Source: Figure 5 · p. 13, Figure 6 · p. 13, Figure 7 · p. 14
  • Across two magic performances and a dashcam clip, ReMem selects the ground-truth anomaly answer while the baseline predicts an incorrect option.The examples describe distinguishing unexpected events from prominent actions or misleading earlier context.Source: Figure 8 · p. 14, Figure 10 · p. 15, Figure 9 · p. 15
  • In table-tennis and basketball sequential-question examples, ReMem tracks referenced entities and updates their scores correctly, unlike the baseline.The questions depend on earlier answers and evolving scores during the same match or play.Source: Figure 11 · p. 16, Figure 12 · p. 16
Loading 2610.05940v1…