PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory as Weights: Internalizing Long-Term History for Streaming Videos

Zhuoming Liu, Jinhong Lin, Lin Zhang, Yiwu Zhong, Yin Li

arXiv:2610.22070v1cs.CVcs.AIcs.LG

TL;DR

Streaming VLM memory must retain history without crowding current observations. MaLoW carries history in evolving LoRA weights while processing current input, and reports gains on streaming benchmarks and VLA tasks.

  • Problem

    Streaming VLMs need to update historical memory incrementally without displacing current observations or continually expanding the input context.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    MaLoW uses a recurrent HyperNetwork to encode history into evolving LoRA weights, leaving the current observation and query as inference inputs.

    Source: §1 · pp. 1–2, §3.1 · p. 4

  • Results

    MaLoW outperforms prior memory approaches by 3.9%–11.8% on OVO-Bench backward split across video backbones, with modest latency overhead, and improves Simpler WidowX action success by 9.3%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MaLoW achieves strong streaming video-understanding results, substantial gains on memory-critical tasks, and transfers its memory mechanism to VLA models.

    Source: §5 · p. 11

    The authors hypothesize that baselines may benefit from fine-tuning their base VLMs, whereas MaLoW keeps base VLMs intact.

    Source: §4.1 · p. 8

Abstract

from arXiv · show

Streaming vision-language models (VLMs) must continuously integrate information from the past while remaining responsive to current observations and user queries. Existing memory mechanisms typically compress, retrieve, or cache historical observations and reintroduce them into the input context, forcing historical memory and current observations to compete for limited context. We propose MaLoW, a streaming memory framework that instead internalizes historical context into evolving lightweight parameter updates. MaLoW uses a recurrent HyperNetwork to incrementally consolidate incoming video observations into fixed-size LoRA weights, allowing historical information to be carried forward without revisiting the full video stream. In doing so, MaLoW represents history as weights, while keeping the current observation as input. To support training, we further construct MaLoW-Instruct, a large-scale streaming video dataset containing 381K question-answer pairs over 56K videos. We conduct extensive experiments to demonstrate that MaLoW achieves strong performance on OVO-Bench, StreamingBench, and StreamArena. On the backward split of OVO-Bench, MaLoW outperforms prior memory approaches by 3.9% to 11.8% across different video backbones while incurring only modest latency overhead. Beyond streaming video understanding, MaLoW also generalizes to vision-language-action models, improving the action success rate on Simpler WidowX by 9.3%.

1 Introduction

Streaming VLMs must update memory incrementally without displacing current observations or adding substantial inference overhead. MaLoW stores history in evolving LoRA weights, and experiments report gains across streaming benchmarks and VLA settings.

1 Introduction

  • Streaming memory must preserve important visual information within bounded capacity, evolve incrementally, avoid crowding current observations, and add minimal inference overhead.Streaming VLMs also need to reason about past events, interpret current events, and remain attentive to what may happen next under latency and memory constraints.Source: §1 · p. 1
  • MaLoW internalizes historical context into evolving LoRA weights while keeping the current observation and user query as inference inputs.The base VLM remains unchanged; only its lightweight LoRA memory evolves over time.Source: §1 · pp. 1–2
  • A recurrent HyperNetwork consolidates incoming observations into fixed-size LoRA memory, using recurrent states to carry history without revisiting the full stream.The method compresses each clip and textual context, updates hidden states through local-temporal and global-semantic branches, then projects them to LoRA weights.Source: §1 · p. 2
  • MaLoW-Instruct provides 381K question-answer pairs over 56K videos for streaming video understanding training.The dataset adapts public offline video-understanding datasets to the streaming setting.Source: §1 · p. 2
  • On OVO-Bench backward tracing, MaLoW outperforms prior memory approaches by 3.9%–11.8% across video backbones, with modest latency overhead.The paper also reports state-of-the-art results on OVO-Bench, StreamingBench, and StreamArena, and a 9.3% action-success improvement on Simpler WidowX.Source: §1 · p. 2

2 Related Work

Prior work addresses long-video and streaming VLM challenges through memory, context management, data, architecture, and video-encoder strategies. MaLoW differs by progressively internalizing incoming visual context into low-rank model representations.

2 Related Work

  • Streaming VLM research also improves instruction-tuning data, streaming architectures, and video encoders; small local video windows perform strongly on real-time questions.The cited local-window findings are reported on existing streaming video benchmarks.Source: §2 · p. 3
  • Offline long-video work explores memory and context-management mechanisms, while streaming memory approaches generally compress history into bounded token budgets.Streaming approaches include textual summaries, token clustering or merging, hierarchical memories, and fixed-size buffers.Source: §2 · p. 3
  • Concurrent work moves long-term memory outside attention into a separate test-time-training branch, whereas MaLoW generates low-rank adaptations directly within the VLM.The cited concurrent method uses online-updated fast weights.Source: §2 · p. 3
  • LoRA has been used to encode data-specific patterns and context, but cited approaches typically need additional optimization for new patterns or retain access to historical context.δ-mem (Lei et al., 2026) dynamically represents textual information through LoRA updates without test-time training, yet still relies on historical context at inference.Source: §2 · p. 3
  • MaLoW targets the streaming setting with a mechanism that progressively incorporates incoming visual context into low-rank model representations.Source: §2 · p. 3

3 Internalizing Long-Term History for Streaming Videos

MaLoW formulates streaming VLMs as using accumulated history and the current observation to answer online, then moves historical context into evolving LoRA weights. Its design updates recurrent memory, trains the HyperNetwork through generated weights, and is supported by MaLoW-Instruct, a dataset of streaming video questions and answers.

3.1 Problem Formulation: Streaming Video and Internalizing History

  • MaLoW defines streaming VLM responses as using historical context, the current observation, and the current query, then represents history in evolving LoRA weights while leaving the current observation as input.The recurrent HyperNetwork incrementally encodes history into LoRA updates, avoiding explicit processing of the full history at inference.Source: §3.1 · pp. 3–4

3.2 Our Design: MaLoW

  • MaLoW compresses video and textual context into q_{t−1}, updates local-dynamics and global-semantics states, and decodes those states into LoRA weights.The local state uses an SSM; the global state uses gated summation, while a similarity-based binary gate controls the latter update.Source: §3.2 · pp. 4–5

3.3 Model Training and Theoretic Analysis

  • Training freezes the base VLM and routes historical context only through generated LoRA weights, optimizing the HyperNetwork to preserve useful history and discard redundancy.The objective combines next-token prediction with regularization of generated LoRA-weight magnitudes; the paper also analyzes the feasibility of representing history-induced transformations with low-rank updates.Source: §3.3 · p. 5

3.4 Training data construction: MaLoW-Instruct

  • MaLoW-Instruct contains 381K instruction samples across 56K videos, with approximately 36% of questions timestamped after their corresponding events end.Its pipeline generates questions and answers and verifies them; the dataset draws from 14 datasets, and question types are approximately evenly distributed among multiple-choice, open-ended, and yes/no.Source: §3.4 · pp. 5–6

4 Experiments and Results

Experiments show MaLoW performs strongly across streaming-video benchmarks and memory-focused evaluations, with additional demonstrations in long-horizon recall and robotic action prediction. The results also examine matched-scale comparisons, latency, and performance across model families.

4.1 Streaming Video Understanding

  • 74.5% on OVO-Bench and 84.8% on StreamingBench are new state-of-the-art scores; MaLoW (Qwen3.5-9B) also leads EPM at 63.6%.The table caption notes that VideoChat3 uses different frame rates on the two benchmarks and that some gray values differ from re-evaluations.Source: Table 1 · p. 7, §4.1 · p. 8
  • 3.9% and 3.5%: at 4B, MaLoW exceeds StreamTTT-4B on OVO-Bench backward and StreamingBench, respectively; at 7–9B, it exceeds OASIS-8B and AURA-8B by 9.2% and 6.0% on OVO-Bench backward.Source: §4.1 · p. 8
  • MaLoW trails StreamTTT-4B and AURA-8B on OVO-Bench real-time; the authors hypothesize that those baselines may benefit from fine-tuning their base VLMs.Source: §4.1 · p. 8
  • 1.4%–4.0% additional parameters: MaLoW keeps the pretrained backbone unchanged; its memory adds around 23 ms, under 15% of total inference time, and adapter generation can run in parallel with backbone inference.Source: §4.1 · pp. 8–9, Figure 4 · p. 9
  • 22.8 average score: on StreamArena HR, MaLoW outperforms VST and MOSS-VL-Realtime by 2.2% and 4.5%, respectively, with a median query-evidence gap of 12.1 minutes.Source: §4.1 · p. 9, Table 3 · p. 9
  • Across 0.8B–4B Qwen3.5 and Gemma4 models, MaLoW improves overall performance, beating full-context baselines on real-time splits and local-window baselines on memory-intensive splits.The broader evaluations also report consistent gains across 4B–11B scales and several offline- and streaming-pretrained VLM families.Source: §4.1 · pp. 8–9, Table 4 · p. 10

4.2 Additional Demonstration and Analysis

  • 59.4% to 68.8% average success rate, a 9.4% gain: MaLoW improves action prediction on Simpler-Bridge with the WidowX robot by internalizing historical observations and action trajectories.Source: §4.2 · pp. 9–10, Table 2 · p. 9
  • Qualitative examples show recall of details from 16, 19, and 29 minutes earlier, updates that avoid obsolete cues, and recovery of relevant information when current visual evidence is occluded.Source: §4.2 · p. 10, Figure 5 · p. 10

5 Conclusion

The section summarizes MaLoW’s approach of carrying historical video context in evolving low-rank weights while keeping the current observation as input, alongside an analysis of how low-rank updates can represent or approximate historical context. It frames this analysis as a feasibility interpretation supporting MaLoW as a learned instantiation, not as evidence of reconstructing the analytical history matrix or matching full-context attention.

5 Conclusion

  • MaLoW stores history in evolving LoRA weights while keeping the current observation as input, leaving the base VLM architecture and parameters unchanged.This separates long-term history from current input without revisiting the full video stream or continually expanding the context window.Source: §5 · p. 11
  • MaLoW achieves strong results across streaming video benchmarks, with gains on memory-critical tasks and modest inference overhead, and improves action success when transferred to VLA models.The conclusion reports these outcomes without specifying benchmark scores or the size of the action-success improvement.Source: §5 · p. 11
  • MaLoW uses a recurrent HyperNetwork to update LoRA weights and is supported by MaLoW-Instruct, a dataset of 381K question-answer pairs over 56K videos.The conclusion presents these as the framework and training-data contributions.Source: §5 · p. 11

A.1 Historical Context as a Read Operator

  • For fixed historical keys and values, the appendix interprets a history read as a shared matrix that encodes how the history responds to queries.The mathematical interpretation does not claim equivalence to full-context attention or explicit reconstruction of the analytical history matrix.Source: A Mathematical Interpretation · p. 16, §A.1 · pp. 16–17

A.2 From Historical Reads to Feature Transformations

  • The historical read changes the feature entering the MLP, expressible as a history-dependent feature transformation.The appendix identifies reproducing the MLP’s processing of this changed feature as the role of an MLP update.Source: §A.2 · p. 17

A.3 Low-Rank Realizations in MLP Projections

  • A rank-r approximation of the history matrix yields rank-at-most-r LoRA-style gate and up projection updates; smaller ranks approximate the read, while sufficient rank can make it exact.The appendix also describes a down-projection construction, whose low-rank condition is separate from low rank of the history matrix.Source: §A.3 · pp. 17–18

A.4 Updating the Historical Effect Online

  • Appending a clip adds its historical effect to the feature transformation, while MaLoW learns a recurrent state and generated adapters as a task-trained surrogate for update-and-compression.The online interpretation assumes previously produced keys and values remain fixed.Source: §A.4 · pp. 18–19

A.5 Extensions to the Backbone Computation

  • The appendix extends the interpretation to normalized kernel reads, where the denominator is part of the historical read, and to softmax reads through a local affine approximation.The softmax approximation is described with an error bound dependent on the query offset and Jacobian approximation.Source: §A.5 · p. 19
  • The derivation accounts for history’s effects on attention mixing, normalization, and earlier-layer outputs, and relates a shared low-rank fit’s residual to output error.The residual measures how well the shared low-rank matrix captures the full history-induced input change; the output-error bound assumes the block is Lipschitz on the relevant segment.Source: §A.5 · p. 20
  • The analysis establishes that history-dependent feature changes can be realized or approximated through LoRA-style updates to Transformer-block MLP projections.It presents this as theoretical feasibility under the stated assumptions, with empirical results showing MaLoW as an effective learned instantiation.Source: §A.5 · p. 20
  • The analysis does not claim that MaLoW exactly reconstructs the analytical history matrix or that its learned state and adapters equal full-context attention.It offers a mathematical interpretation of parameter-space history representation rather than an equivalence result.Source: §A.5 · p. 20

B Additional Dataset Details

MaLoW-Instruct is assembled from curated existing video collections, with overlap filtering to reduce evaluation contamination. The resulting corpus contains 55,867 unique videos from 14 underlying datasets.

B Additional Dataset Details

  • The dataset construction section covers video sources, the data pipeline and annotation construction, and dataset statistics.Table 5 provides a distribution of training videos across underlying datasets and data sources.Source: B Additional Dataset Details · p. 20, Table 5 · p. 21

B.1 Video Sources

  • 55,867 unique videos from 14 underlying datasets remain after aggregating sources and removing duplicates and evaluation overlaps.Videos overlapping with OVO-Bench are filtered; the text says StreamingBench and StreamArena do not overlap with the video sources.Source: §B.1 · p. 21
  • Sources include TimeLens-100K, the academic subset of LLaVA-Video-178K, ET-Instruct, YouCook2, CrossTask, and THUMOS14.The authors primarily build on existing large-scale collections for scalability and reliability.Source: §B.1 · p. 21

B.2 Data Pipeline and Annotation Construction

The pipeline creates timestamped questions about annotated events, generates answers using only video evidence available at each timestamp, and verifies and filters the resulting pairs before training.

Question Generation

  • Questions target annotated events and use sampled timestamps classified as past, real-time, or future relative to the event interval.Question types include open-ended, multiple-choice, Yes/No, and counting; future questions may require anticipation or abstention when evidence is insufficient.Source: §B.2 · p. 21

Answer Generation

  • Answers are generated using only video content observable up to the question timestamp, avoiding access to later visual information.The answer generator is Qwen3.5-122B-A10B.Source: §B.2 · p. 22

Verification

  • A verifier checks question validity and grounding, and whether the model answer agrees with available evidence, using the expected answer and event description as context.The event description helps distinguish an ambiguous or unreasonable question from an incorrect model answer.Source: §B.2 · p. 22

Filtering and Post-Processing

  • Pairs judged inconsistent, incorrect, invalid, ambiguous, or unreasonable are discarded; retained questions at the same video timestamp are grouped into multi-turn conversations.Offline video-understanding annotations from LLaVA-Video-178K are also included.Source: §B.2 · p. 22

B.3 Dataset Statistic

MaLoW-Instruct contains 381K instruction samples across 56K videos, with questions spanning temporal classes and formats. Questions average 70.4 tokens, while answers average 3.8 tokens.

B.3 Dataset Statistic

  • 381K instruction samples span 56K videos, with real-time questions at 46.1%, past questions at 36.4%, and future questions at 17.4%.Real-time questions occur during the target event, past questions after it, and future questions before it.Source: §B.3 · p. 22
  • Multiple-choice questions account for 35.0%, open-ended for 34.7%, Yes/No for 30.0%, and counting questions for 14.0%.Table 7 summarizes question-type distribution.Source: §B.3 · p. 22
  • Questions average 70.4 tokens and answers 3.8 tokens; question video length averages 57.2 seconds, compared with 89.5 seconds for full videos.Full videos reach up to 1,673.4 seconds; Figure 6 presents distributions of questions, answers, and video durations.Source: §B.3 · p. 22, Table 8 · p. 23, Figure 6 · p. 23

C.1 Main Experiments

The section details the main benchmark setup for MaLoW, including its video-history preprocessing, base models, and evaluation protocols. It also reports a discrepancy between reproduced and paper-reported StreamingBench results for some baselines.

C.1 Main Experiments

  • The main experiments use four base VLMs: VideoChat3-4B, Qwen3-VL-4B, Qwen3.5-9B, and MOSS-VL-Realtime-11B.They include video-centric and general-purpose models, with full self-attention, hybrid attention, or cross-attention architectures.Source: §C.1 · p. 23
  • MaLoW uses the most recent 2-second block as the current observation and converts earlier frames into LoRA weights; Qwen3-VL-4B instead uses an 8-second current window.Frames are sampled at 1 fps and divided into non-overlapping two-frame blocks; longer current windows improved the reported performance for Qwen3-VL-4B.Source: §C.1 · p. 23
  • A four-layer Transformer aggregator, four-layer DeltaNet state-space model, and converter generate LoRA updates from temporal-block representations and recurrent states.The vision encoder and adaptor are drawn from the corresponding VLM; the generated LoRA is applied to selected modules while the base model remains frozen.Source: §C.1 · p. 24
  • The evaluation covers StreamingBench’s Real-Time split and OVO-Bench’s Real-Time and Backward splits, with OVO-Bench also reporting macro-averages across tasks and splits.The OVO-Bench Backward split includes Episodic Memory, Action Sequence Identification, and Hallucination Detection; HLD is less discriminative because every ground-truth answer is “Unable to answer.”Source: §C.1 · pp. 24–25
  • For MOSS-VL-Realtime-11B, VideoChat3-4B, and Qwen3.5-9B, the authors re-evaluate the baselines; remaining baseline results come from their original papers.The reproduced OVO-Bench results for VideoChat3 and MOSS-VL match reported performance, but reproduced StreamingBench results are lower, so both result sets are included.Source: §C.1 · p. 25

C.2 MaLoW in VLA setting

The appendices describe MaLoW’s VLA adaptation, additional base-model evaluation settings, and a historical-retrospection stress test. They specify action-history inputs and evaluation protocols, but the supplied passages do not report the VLA success-rate comparison or stress-test outcomes.

C.2 MaLoW in VLA setting

  • The VLA setup uses Q3.5-2B-MLP as its starting model, freezes the merged Qwen backbone, and trains MaLoW components together with the action decoder.The action objective is combined with a LoRA regularization term; the base policy was trained on Bridge and RT-1 data and reached 59.4% on Simpler-Bridge with WidowX.Source: §C.2 · pp. 25–27
  • The VLA variant pairs each historical frame with its preceding action and incorporates visual, timestamp, and action-history tokens into the aggregator.It is evaluated interactively on four WidowX Bridge tasks, with 24 episodes per task; its base policy exactly matches Q3.5-2B-MLP at environment step zero.Source: §C.2 · pp. 26–27

C.3 MaLoW with different VLM scales and sources

  • For the additional model-scale evaluation, MaLoW uses Gemma-E4B and Qwen3.5-0.8B, 2B, and 4B, with rank-8 LoRA applied to specified attention and MLP modules.OVO-Bench is evaluated on backward, real-time, and forward splits; StreamingBench evaluation includes Real, Omni, Sequential-QA, and Proactive subsets.Source: §C.3 · pp. 27–28

C.4 Memory stress test

  • The StreamArena Historical Retrospection stress test contains 877 questions across four evidence-to-query gap levels and 237 videos.The comparison includes full-history, local-window, and text-memory approaches; MaLoW uses the final two seconds as current input and converts earlier frames into LoRA.Source: §C.4 · p. 28

C.5 Streaming Inference Speed Profiling

The profiling section measures synchronized GPU latency for streaming inference across MaLoW and four baselines on sampled OVO-Bench questions spanning varied video durations. It describes what computation each method’s latency includes, but gives no comparative latency outcomes in the supplied passages.

C.5 Streaming Inference Speed Profiling

  • Latency is profiled on 100 OVO-Bench questions with video durations from 8 to 1,117 seconds, excluding decoding, resizing, and other input preprocessing.The sample has a median duration of 153.5 seconds, a mean of 193.3 seconds, and a 90th percentile of 312 seconds.Source: §C.5 · p. 28
  • MaLoW’s measured streaming step includes current-observation encoding, hidden-state and adapter updates, model prefilling, and first-token decoding.The prefill uses the question and two recent frames from the current two-second window.Source: §C.5 · p. 28
  • The baseline latency protocols cover recent-window SimpleStream and VideoChat3, full-history Qwen3.5, and MOSS-VL’s growing vision-text KV state.VideoChat3 re-prefills a sliding window of 32 frames, while full-context Qwen3.5 prefills the complete history sampled at 1 fps.Source: §C.5 · pp. 28–29

D.1 Ablation Study

The ablations examine LoRA placement and rank, MaLoW’s local-temporal and global-semantic components, global-semantic aggregation, current-observation size, and hidden-state refresh rate. The results describe trade-offs between memory and real-time perception and identify the best-performing tested configurations.

D.1 Ablation Study

  • Extending LoRA from shared MLP modules to self-attention v_proj and linear-attention qkv modules consistently improves average performance, with all three module types performing best.The comparison cumulatively adds LoRA modules in Qwen3.5-9B’s hybrid attention architecture.Source: §D.1 · p. 29
  • Rank 8 achieves the best overall performance, with an average score of 69.2; rank 16 causes a substantial performance drop on both benchmarks.Rank 4 performs slightly worse, while rank 16’s drop is attributed by the authors to overfitting.Source: §D.1 · p. 29
  • Combining the SSM and global semantic aggregation yields the best overall performance, balancing real-time understanding with long-term memory.The SSM variant performs better on StreamingBench, while the sum variant performs better on Episodic Memory.Source: §D.1 · p. 29, Table 11 · p. 30
  • Gated sum achieves the best average performance among the tested global-semantic aggregation strategies by filtering redundant representations and retaining informative observations.The alternatives are direct sum and cosine-similarity-weighted sum.Source: §D.1 · p. 29, Table 12 · p. 30
  • A 4-second current-observation window improves real-time perception but lowers EPM to 60.0, while a 1-second window reduces real-time perception and gives similar EPM to the 2-second window.The authors use a 2-second window by default to balance real-time perception and long-term memory.Source: §D.1 · p. 30, Table 13 · p. 30
  • Performance remains stable across hidden-state update intervals; less frequent updates slightly improve memory performance at the cost of a small real-time-perception drop.The authors adopt a 2-second interval to balance the two capabilities and match the current-observation window.Source: §D.1 · p. 30, Table 14 · p. 30

D.2 Additional Analysis

MaLoW maintains real-time perception as historical context grows, while adding modest inference latency. Its LoRA state changes alongside video-event surprise, though some long-range memories remain coarse and agentic tasks are outside its training scope.

Latency Breakdown

  • 1.75 ms of added critical-path latency raises Qwen3.5-9B inference latency from 130.07 ms, about 1.3% of effective end-to-end latency.Adapter preparation can be pipelined with current-block inference; Table 15 reports latency percentages relative to a 152.89 ms total.Source: §D.2 · p. 30, Table 15 · p. 31

LoRA Hidden State Evolution over the History

  • The LoRA state changes more rapidly during high-surprise video intervals and more smoothly during the lower-surprise interval.The reported high-surprise periods are before 60 seconds and after 130 seconds; the lower-surprise interval is 60–130 seconds.Source: §D.2 · p. 31, Figure 7 · p. 31

Effect of Memory on Real-Time Perception

  • Across OVO-Bench and StreamingBench, MaLoW maintains stable real-time perception across varying history lengths.On OVO-Bench, it matches the local-window baseline across prefixes up to 28 minutes; on StreamingBench, accuracy remains stable as history extends from 5 to 60 minutes. The authors attribute this consistency to regularization of generated LoRA-weight norms.Source: §D.2 · p. 31, Figure 8 · p. 32

Additional Qualitative Results

  • After a clue 32 minutes earlier, MaLoW retained a coarse Coca-Cola memory and answered “beverage” rather than naming the specific object.The authors describe this as an example of retained memory being vague.Source: §D.2 · p. 32
  • Tasks requiring revisiting frames with agentic tools remain outside MaLoW’s training scope, while fine-grained visual perception may also limit performance.The authors leave extension to agentic settings for future work and expect stronger base-VLM perception to benefit certain cases.Source: §D.2 · p. 32
Loading 2610.22070v1…