FlashBack: Knowing When to Remember in Streaming Vision-Language Models

1 Introduction

• Streaming VLMs must answer questions about both current scenes and earlier events while processing growing video streams with limited computation and working memory.Live commentary and visual assistance require answering from only the frames seen so far.Source: §1 · p. 1
• Keeping only recent video content can lose earlier details, whereas adding history can make answers about the current scene less accurate.The introduction illustrates the trade-off with recalling where a red cup was placed versus identifying a blue bottle being held.Source: §1 · p. 1
• Prior memory approaches address how to store and retrieve history, while FlashBack adds a decision about whether historical memory should participate at all.The surveyed approaches compress or structure history and use direct, selective, adaptive, or agentic retrieval.Source: §1 · pp. 1–2
• FlashBack uses a frozen VLM to decide whether a query needs historical evidence, then routes inference to Native or Recall before retrieval.Recall combines historical evidence with recent context through query-local Side-KV; its temporary KV states are discarded rather than written to persistent Native state.Source: §1 · p. 2
• FlashBack improves several long-horizon and memory-dependent tasks across two streaming VLMs and two benchmarks while largely preserving real-time perception.It is training-free and uses Side-KV to keep recalled history isolated from the Native streaming state.Source: §1 · p. 2

2 Related Work

• Conventional VLMs are made suitable for continuous inputs through sliding windows or surrounding memory and interaction mechanisms, rather than persistent causal state reuse.The approaches make continuous inputs tractable but largely rely on inference or adaptation procedures around finite-image or finite-clip pipelines.Source: §2 · p. 3
• Native streaming models reuse online state: StreamingVLM maintains bounded KV state, while Mage-VL-4B uses codec-aware incremental encoding and a dual-system architecture.FlashBack is instantiated on both models to evaluate it beyond one streaming architecture.Source: §2 · p. 3
• Streaming-video memory systems compress or manage KV caches, retrieve stored representations, organize higher-level memories, or use agentic search and tools.These approaches address how history is maintained and how stored evidence is brought back into reasoning.Source: §2 · p. 3
• Recent methods selectively retain, retrieve, or assemble history, whereas FlashBack first decides whether memory-augmented reasoning should be invoked.It maps each query's temporal-evidence requirement to Native or Recall before retrieval, treating history as conditional evidence.Source: §2 · p. 3

3.1 Overview

• FlashBack preserves the Native trajectory and uses a temporal-evidence router to decide whether a query should remain Native or invoke Recall before history is accessed.This separates the decision to recall from the later selection of historical evidence.Source: §3.1 · p. 3, §3.3 · p. 4

3.2 Causal Multi-level Memory

• The online causal bank pairs recent backbone-native visual states with completed historical units, keeping local continuity and distant evidence available without placing history in the active reasoning context by default.The bank is built only from the observed video prefix; recent states move with the streaming boundary as completed units accumulate in long-term memory.Source: §3.2 · pp. 3–4

3.3 Knowing When to Remember

• The router maps frozen-VLM temporal-evidence semantics to Native or Recall before accessing history, with five semantic states distinguishing current, historical, relational, distributed, and predictive evidence needs.Current queries receive a further assessment that distinguishes directly observable boundary or ongoing evidence from evidence established earlier.Source: §3.3 · pp. 4–5

3.4 Recall through Side-KV

• Recall retrieval uses a question-conditioned prefilter and query–key reranking over completed historical units, without future observations or answer correctness.The retrieved units are combined with recent context in a query-local Side-KV pathway that is discarded after the answer, leaving persistent Native state unchanged.Source: §3.4 · pp. 5–6

4.1 Implementation Details

• The main experiments use StreamingVLM and Mage-VL-4B on OVO-Bench and StreamingBench; component ablations use StreamingBench-MA and OVO-Bench-CRR.The finite-answer evaluation scores allowed candidates using next-token logits after a fixed answer prefix.Source: §4.1 · p. 6, §4.2 · p. 6

4.3 Ablation Study

• Structured routing preserves Recall's CRR gain and further improves MA, whereas direct binary routing loses the CRR improvement and approaches the Native baseline.Recall alone produces a large CRR gain but only modestly changes MA.Source: §4.3 · p. 6, Table 1 · p. 6
• Adding one long-term historical page improves CRR, but retrieval budgets above K = 1 add no CRR gain and slightly reduce MA at K = 5.The study uses K = 1 as the default for compact access to long-term evidence.Source: §4.3 · p. 6, Table 2 · p. 6
• Side-KV improves both MA and CRR over other memory interfaces, with a particularly large margin on CRR.The compared interfaces use the same selected historical page, indicating that how history is incorporated also matters.Source: §4.3 · pp. 6–7, Table 3 · p. 7

4.4 Main Results

• FlashBack improves representative long-range temporal tasks while largely preserving Native real-time performance across the reported results.On OVO-Bench, gains are clearest on Forward reasoning; on StreamingBench, Real-Time Visual Understanding improves on both backbones.Source: Table 4 · p. 7, Table 6 · p. 7, §4.4 · pp. 7–8
• FlashBack's most consistent OVO-Bench gains are on REC and CRR, while EPM and ASI improvements depend more on the backbone.EPM is unchanged on StreamingVLM and improves slightly on Mage-VL.Source: §4.4 · p. 7
• StreamingBench results vary by task group: Real-Time Visual Understanding improves on both backbones, but Mage-VL declines on Omni-Source and Contextual Understanding.Some tasks require accumulating information over the stream, while others require suppressing stale history.Source: §4.4 · pp. 7–8
• FlashBack increases processing time relative to Native, yet both backbones process streams faster than they arrive at the evaluated 1 and 2 FPS input rates.The reported throughput leaves processing headroom under both settings.Source: §4.4 · p. 8, Figure 4 · p. 8

5 Conclusion

• Across two streaming backbones and two benchmarks, FlashBack improves multiple tasks requiring historical recall and long-range temporal evidence while largely preserving Native real-time capability.It achieves competitive performance without additional training.Source: §5 · p. 9
• FlashBack selectively accesses long-term history through temporal-evidence routing and an isolated Side-KV Recall trajectory.Source: §5 · p. 9
• The framework is training-free.Source: §5 · p. 9

6 Limitations

• Neither the temporal-evidence router nor the memory retriever is explicitly optimized for downstream streaming tasks.FlashBack is training-free.Source: §6 · p. 9
• Historical access supports long-range evidence queries but can interfere with hallucination-sensitive and stale-context-filtering tasks.Source: §6 · p. 9
• Future work may use uncertainty- and validity-aware routing and retrieval to suppress unsupported or outdated evidence while retaining the isolated Recall pathway and frozen-backbone setting.Source: §6 · p. 9

A.1 Component Ablation Settings

• Ablations vary the memory or routing component while holding video sampling, query inputs, answer prefix, candidate space, and scoring rule fixed.The diagnostic subsets are StreamingBench Multimodal Alignment and OVO-Bench Clues Reveal Responding.Source: §A.1 · p. 11

A.3 Answer Scoring

• Finite-answer tasks use next-token candidate logits after a fixed answer prefix, with Native and ablations sharing candidate coordinates and tie-breaking.OVO-Bench uses a fixed 17-token space, while StreamingBench uses each question’s option labels; experiments use visual input only.Source: §A.1 · p. 11, §A.3 · p. 11

A.2 Main Experiment Settings

• Within each backbone–benchmark pair, Native and FlashBack share video sampling, query inputs, and answer-scoring settings; cross-backbone FPS rates are not a controlled comparison.StreamingVLM uses 4-second units and 16-second recent context; Mage-VL-4B uses 4-frame units and 16-frame recent context.Source: §A.2 · p. 11

A.4 Temporal-Evidence Routing

• Structured routing first classifies temporal evidence as Current, Historical, Interaction, Aggregation, or Predictive, then refines Current requests into four states.The semantic states are inference-time variables, not supervised labels, and no external routing annotations are used.Source: §A.4 · p. 12
• Fixed mappings send Historical, Interaction, and Aggregation to Recall; Current Boundary and Ongoing requests stay Native, while Persistent and Outcome requests invoke Recall.Predictive requests remain on Native; when no eligible history exists, Recall uses causally available recent context.Source: §A.4 · p. 13

A.5 Trajectory-Selection Analysis

• The analysis retains fixed Native and Recall outputs and evaluates routing by downstream answer correctness rather than manually defined routing labels.Both-correct queries count as successful; Neither queries are excluded because neither fixed output is correct.Source: §A.5 · pp. 14–15
• Structured Routing selects a correct trajectory for more than 92% of answerable queries on both MA and CRR.It substantially improves unique-correct-trajectory selection over Direct Routing on both subsets, with the largest difference on CRR.Source: Table 8 · p. 14, §A.5 · p. 15
• Structured Routing falls 6.00 percentage points below the Trajectory Oracle on MA and 4.58 points below it on CRR.The Oracle is an analytical upper bound based on whether either fixed trajectory output is correct, not an inference-time policy.Source: §A.5 · p. 15

A.6 Long-Term Retrieval Implementation

• Eligible history consists of completed pages preceding the recent context, and FlashBack maintains this causal memory bank online from the observed video prefix.The implementations retain native-representation history and materialize full-layer page KV for selected pages as needed, without a trained retrieval head.Source: §A.6 · pp. 15–16
• A frozen, question-conditioned prefilter narrows historical pages to a candidate set before decoder-level reranking.Candidates are then scored using the frozen decoder’s query–key space; the score length-normalizes a log-sum-exp over page visual keys and averages across heads and retrieval layers.Source: §A.6 · p. 15

A.7 Side-KV Implementation and Hard-Insertion Controls

• Recall combines a retrieved historical page and recent context in an isolated, query-local trajectory, leaving the persistent Native state unchanged.The temporary decoder state is released after the query; subsequent streaming continues from Native.Source: §A.7 · p. 16
• Hard-insertion controls compare historical, zero, and random KV content against Side-KV using the same selected page, token count, positions, attention span, query, and answer scorer.When no historical page is eligible, all variants use the same recent-context fallback.Source: §A.7 · p. 16

A.8 Efficiency Measurement Protocol

• Timing covers two repetitions of 26 queries over 1046.5 seconds of video on one A100 GPU, at 1- and 2-FPS sampling.FlashBack timing includes memory construction, routing, reranking, Recall when invoked, and answer-logit computation; throughput is distinct from per-query latency.Source: §A.8 · p. 17

A.9 Memory-Bank Growth and Computational Complexity

• Archived history grows with observed video duration, and the reported configuration neither imposes a capacity limit nor evicts completed units.Under fixed sampling and page settings, storage grows approximately linearly; materialized page caches may add storage.Source: §A.9 · p. 17
• Semantic prefiltering scores eligible historical units, while more expensive decoder-level reranking and page materialization are restricted to shortlisted candidates.The candidate set is formed from top-ranked units for each query hypothesis, with duplicates removed.Source: §A.9 · p. 17

A.10.1 Visualization Examples

• In two action-sequence examples, Recall retrieves earlier preparation or grilling events that answer questions about what happened before a later action.The retrieved intervals are 157.7–161.7 s for dough preparation and 152–156 s for placing steak on a grill; one Native answer selects an unrelated event.Source: §A.10.1 · p. 18
• In the contrasting vehicle example, Native identifies a Sedan while Recall predicts SUV despite retrieving a page containing the queried car.The passage attributes the failure to using the evidence for a finer-grained vehicle-type decision.Source: §A.10.1 · p. 18, Figure 7 · p. 18
• The examples distinguish retrieving relevant history from benefiting from it: the final prediction still depends on how the frozen backbone interprets the evidence.The visualizations pair retrieved pages and their temporal locations with Native and Recall answers.Source: §A.10.1 · p. 18

• Table 9 reports FlashBack’s detailed task-level results on OVO-Bench.Source: Table 9 · p. 19, §B.1 · p. 19
• The table headers organize results into Realtime, Backward, Forward, and Overall categories.Source: §B.1 · p. 19
• The listed task and metric headers include OCR, ACR, ATR, STU, FPD, OJR, EPM, ASI, HLD, REC, SSR, and CRR.Source: §B.1 · p. 19

• Tables 10 and 11 report detailed task-level results on StreamingBench.Source: §B.2 · p. 19, Table 11 · p. 19
• Table 10 covers Real-Time Visual Understanding results on StreamingBench.Source: Table 10 · p. 19
• Table 11 presents detailed StreamingBench results for FlashBack.Source: Table 11 · p. 19