PaperlayerView exact revision on arXiv

Source-linked AI summary

When to Retrieve, When to Stay: Uncertainty-Aware Temporal Evidence Allocation for Streaming Video-LLMs

Xiang Hu, Jiazuo Yu, Lu Zhang, Yunzhi Zhuge, Huchuan Lu

arXiv:2609.37345v1cs.CV

TL;DR

Streaming Video-LLMs must choose useful evidence from a growing history using only observations available by query time, under a bounded visual-processing budget. WRWS adapts semantic retrieval and recency using retrieval uncertainty, then encodes only selected observations with the target model. Across multiple model families and scales, it achieves competitive or improved online performance while reducing VTAT to 47.93% of SimpleStream.

  • Problem

    Streaming Video-LLMs must select useful observations from a growing history using only causally available evidence under a bounded target-model visual-processing budget.

    Source: §1 · p. 1

  • Method

    WRWS uses lightweight external representations and retrieval uncertainty to balance semantic relevance with recency, selecting evidence before target-model visual encoding.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    47.93% of SimpleStream VTAT, with comparable downstream performance across multiple Video-LLM families and model scales.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    The query-level choice between semantic evidence and recent context can be made before invoking costly target-model visual encoding.

    Source: §1 · p. 2, §3.1 · p. 4

    Cached lightweight representations grow with stream length; compression or eviction could improve scalability to very long streams.

    Source: H Limitations and Future Work · p. 20

Abstract

from arXiv · show

Streaming video understanding requires Video Large Language Models (Video-LLMs) to reason over continuous visual streams under causal constraints. As the visual history grows, a bounded visual?processing budget requires evidence selection that balances temporal recency with query relevance. Recent-only selection excludes potentially relevant historical evidence, whereas Semantic-only retrieval can displace useful recent context when relevance scores are ambiguous. We introduce WRWS (When to Retrieve, When to Stay), a training-free framework for uncertainty-adaptive evidence allocation. A lightweight external vision-language encoder scores query relevance across the observed history, while an adaptive allocation module uses the normalized entropy of the similarity distribution as a proxy for retrieval uncertainty. WRWS favors semantic retrieval when relevance cues are reliable and strengthens the recency prior under uncertainty. Following a retrieve-first, encode-later pipeline, WRWS selects evidence before target-model visual encoding, such that only the selected observations are processed by the costly target Video-LLM. Experiments across four Video-LLM families and multiple model scales demonstrate competitive accuracy on StreamingBench and OVO-Bench. In our efficiency evaluation, WRWS reduces average vision-to-answer time to 47.93% of the state-of-the-art method. Code will be released.

1 Introduction

Streaming Video-LLMs must select causally available observations for a query under a bounded target-model visual-processing budget. WRWS uses retrieval uncertainty to adapt the balance between semantic relevance and recency, selecting evidence before costly target-model encoding.

1 Introduction

  • Streaming queries must use only observations available by query time, while selecting relevant evidence from an increasingly long history under a bounded visual-processing budget.Recent-only selection can exclude useful earlier evidence, whereas semantic-only retrieval can displace useful recent context when similarities are ambiguous.Source: §1 · pp. 1–2
  • WRWS uses lightweight external representations and retrieval uncertainty to adaptively balance semantic relevance against recency before invoking the target Video-LLM on selected evidence.The model-external design requires neither parameter updates nor access to target-model hidden states, attention statistics, or historical KV caches.Source: §1 · p. 2
  • Across four Video-LLM families and multiple model scales, WRWS achieves competitive or improved online performance while reducing VTAT to approximately 48% of SimpleStream.The evaluation also includes offline long-video understanding.Source: §1 · pp. 2–3
  • Query-level complementarity between Recent-only and Semantic-only motivates choosing how strongly semantic relevance should override recency for each query.WRWS selects Top-K evidence before target-model encoding, keeping expensive visual processing restricted to the selected observations.Source: §1 · p. 2

2 Empirical Observations

Profiling shows that target-model visual processing accounts for a substantial part of vision-to-answer time. Fixed recency and semantic policies have complementary query-level strengths, motivating adaptive evidence allocation.

2.1 Do We Need to Process Every Incoming Observation with the Target VLM?

  • 12.18 s versus 0.243 s (Qwen2.5-VL-7B) and 8.01 s versus 0.317 s (Qwen3-VL-8B): VTAT is 50.1× and 25.3× inference TTFT, respectively.The profiling protocol and runtime breakdown indicate that target-VLM visual encoding is the largest component of VTAT.Source: §2.1 · p. 3

2.2 Is a Fixed Evidence Policy Sufficient?

  • 55.73 versus 48.46 (Qwen2.5-VL-7B) and 58.10 versus 51.76 (Qwen3-VL-8B) are the respective Recent-only and Semantic-only scores.Recent-only's advantage is especially pronounced on Real-Time tasks.Source: §2.2 · p. 3
  • 62.20 (Qwen2.5-VL-7B) and 65.75 (Qwen3-VL-8B) oracle scores exceed Recent-only by 6.47 and 7.65 points, respectively.The per-query oracle gains are consistent across all three task groups.Source: §2.2 · p. 3
  • The per-query oracle gap reveals complementarity between fixed policies, motivating a query-time choice between semantic retrieval and recent context.WRWS adapts that trade-off over causally available history under a fixed Top-K budget, encoding only selected observations with the target Video-LLM.Source: §2.2 · p. 4, Figure 3 · p. 4

3 Method

WRWS allocates a bounded visual-processing budget across causally available observations, using lightweight retrieval to balance query relevance and recency. It selects evidence before invoking the target Video-LLM, which processes only the selected observations for answering.

3.1 Problem Setup

  • For each query, WRWS selects at most K observations from the causally available history for target-model visual processing, while unselected observations bypass that encoding.Future observations are unavailable, and the selected observations are combined with the query for answer generation.Source: §3.1 · p. 4

3.2 Lightweight Query--Observation Retrieval

  • A lightweight vision-language encoder represents arriving observations and the query, then cosine similarity scores their semantic relevance.Scores are calibrated over the available history; the resulting bounded values support uncertainty estimation and semantic–recency allocation.Source: §3.2 · p. 5

3.3 Uncertainty-Aware Temporal Evidence Allocation

  • Normalized entropy of a distribution over calibrated relevance scores estimates retrieval uncertainty, with a query-specific adaptive temperature scaling the softmax to score dispersion.When relevance scores are effectively indistinguishable, the distribution is uniform; for a singleton history, its sole observation has probability 1.Source: §3.3 · p. 5
  • A peaked relevance distribution gives more weight to semantic relevance, while diffuse relevance strengthens recency; recent observations receive larger temporal weights, but earlier ones remain selectable when semantically supported.The uncertainty value α_t continuously interpolates between the semantic and recency signals; for a singleton history, α_t is set to 0 by convention.Source: §3.3 · pp. 5–6

3.4 Budgeted Selection and Delayed Target-VLM Encoding

  • WRWS selects the top K_t observations by evidence score, with K_t=min(K,N_t), then passes them with the query to the target model.When N_t≤K, all available observations are selected; target-model processing is limited to K observations per query, and retrieval requires no target-model states.Source: §3.4 · p. 6

4 Experiments

Experiments evaluate WRWS on streaming and offline video benchmarks across model families and scales. WRWS remains competitive or improves accuracy while substantially reducing vision-to-answer time.

4.1 Experimental Setup

  • Streaming evaluation uses official protocols, causal access to observations, and SigLIP for lightweight retrieval.The benchmarks include StreamingBench, all 12 OVO-Bench tasks, EgoSchema, Video-MME, and MLVU.Source: §4.1 · p. 6

4.2 Main Results

  • Across four Qwen2.5/3 backbones, WRWS matches or improves OVO-Bench averages over SimpleStream while remaining comparable on StreamingBench.On LLaVA-OV-7B, WRWS scores 75.35 on StreamingBench and 56.44 on OVO-Bench, outperforming the reported HERMES variants.Source: §4.2 · p. 7, Table 2 · p. 7
  • WRWS applies without additional training across LLaVA-OV, Qwen2.5-VL, Qwen3-VL, and Qwen3.5 models from 0.5B to 35B-A3B.The reported results span StreamingBench and OVO-Bench; the evaluation also includes offline video benchmarks.Source: §4.1 · p. 6, §4.2 · p. 7
  • On LLaVA-OV-7B, WRWS scores 71.96 on MLVU and a VideoMME average of 58.85, highest among compared training-free methods.It remains competitive on EgoSchema and VideoMME.Source: §4.2 · pp. 7–8

4.3 Efficiency

  • 47.93% of SimpleStream’s VTAT is required on average by WRWS, with relative VTAT decreasing as stream length grows.Savings are larger under the higher visual-token load; the comparison uses Qwen2.5-VL-7B.Source: §4.3 · p. 8

5 Understanding Adaptive Evidence Allocation

The relative utility of Recent-only and Semantic-only varies across temporal tasks. WRWS largely preserves recent evidence but makes larger, more semantically aligned departures when retrieval uncertainty is lower.

5.1 Task-level Behavior

  • Recent-only gains 8.6–27.5 percentage points on all six Real-Time Visual Perception tasks, while Semantic-only performs better on EPM, ASI, and REC.HLD favors Recent-only, whereas CRR and SSR show only modest differences.Source: §5.1 · p. 8

5.2 Sample-level Behavior

  • WRWS matches Recent-only exactly on 79.9% of queries, retaining a strong recency prior.The analysis compares evidence selected by WRWS, Recent-only, and Semantic-only.Source: §5.2 · p. 8
  • From the lowest- to highest-uncertainty quintile, departure magnitude falls from 26.7% to 18.9%, and semantic replacement share falls from 42.1% to 12.2%.Queries are grouped into retrieval-uncertainty quintiles independently within each task.Source: §5.2 · p. 9

6 Ablation Studies

Ablations examine adaptive allocation, the recency prior, uncertainty measures, and lightweight retrieval encoders. The reported comparisons favor adaptive allocation, retaining a recency prior, entropy, and external lightweight representations.

Adaptive allocation

  • Adaptive allocation outperforms the best fixed setting by 0.25 points on Qwen2.5-VL and 0.34 points on Qwen3-VL.The best fixed α differs by backbone: 1.0 for Qwen2.5-VL and 0.75 for Qwen3-VL.Source: §6 · p. 9

Recency prior

  • Removing the recency prior reduces the score from 58.59 to 51.76, while linear and exponential weighting differ by only 0.28 points.The default uses the simpler linear weighting.Source: §6 · p. 9

Retrieval uncertainty

  • Entropy and standard deviation outperform maximum confidence and margin; entropy achieves the best overall result and is the default.The comparison favors measures that characterize the similarity distribution as a whole over measures based only on top-ranked scores.Source: §6 · p. 9

Lightweight retrieval encoding

  • SigLIP achieves the highest lightweight-encoder score, 58.59, while target-model representations score 57.56 and require its expensive visual encoder.The lightweight encoders perform similarly, supporting retrieval before target-model visual processing.Source: §6 · p. 9

7 Related Work

Prior streaming Video-LLM work includes streaming-specific training and architectures as well as training-free management of historical representations and computation. WRWS instead selects causally available observations for target-model visual processing before that processing occurs.

Streaming Video Understanding with Video-LLMs

  • Prior methods manage context growth by organizing or compressing memory and KV states, filtering observations, or reusing visual representations within target-model processing.Other approaches use streaming-specific training and architectures.Source: §7 · p. 10

Training-Free Temporal Evidence Selection

  • SimpleStream retains recent frames, while OASIS triggers retrieval using preliminary MLLM reasoning and ShallowStream routes using target-model logits, KV states, and attention.These approaches differ in how they access historical evidence.Source: §7 · p. 10
  • WRWS uses lightweight external representations and retrieval uncertainty to adjust the semantic–recency trade-off before target-model visual encoding.Its selection is model-external and requires neither preliminary MLLM reasoning nor target-model attention or historical KV states.Source: §7 · p. 10

8 Conclusion

WRWS uses retrieval uncertainty to choose between semantically relevant history and recent context, selecting evidence before target-model visual encoding. The supplementary material examines its empirical basis, further evaluations, efficiency, adaptive allocation, failure cases, and limitations.

8 Conclusion

  • WRWS uses retrieval uncertainty to choose between semantic history and recent context, selecting evidence before target-model visual encoding and reducing average VTAT to 47.93% of SimpleStream.The conclusion reports comparable downstream performance across multiple Video-LLM families and model scales without additional training.Source: §8 · p. 10

A Overview

  • The supplement covers detailed setup, empirical analyses, additional online and efficiency evaluations, adaptive allocation, and qualitative failure cases.It includes VTAT breakdowns and per-task results for fixed policies and a per-query oracle, plus comparisons across budgets, models, and serving backends.Source: A Overview · p. 13
  • The authors identify retrieval-cache growth with stream length and reliance on uncertainty estimated from external retrieval representations as limitations and future-work topics.Source: A Overview · p. 13

B Detailed Experimental Setup

Experiments use eight target Video-LLMs, causal streaming inference, and family-specific evidence budgets. A latency breakdown finds target-model visual encoding is the largest component of VTAT under the profiled setup.

B Detailed Experimental Setup

  • The setup evaluates eight target Video-LLMs, sampling streaming observations at 0.5 fps for LLaVA-OneVision and 1 fps for Qwen families; budgets K=16, 4, and 6 keep target visual tokens at approximately 3K.Streaming K is 16 for LLaVA-OneVision, 4 for Qwen2.5-VL, and 6 for Qwen3-VL and Qwen3.5. Offline LLaVA-OneVision uses K=32, approximately 6K target-model visual tokens, and identical constant temporal weights.Source: B Detailed Experimental Setup · p. 13
  • Inference uses each model’s native processor with greedy decoding up to 256 tokens; main accuracy and efficiency results use standard Transformers inference.Latency is measured on NVIDIA 4090 GPUs; VTAT runs from the first observation to the first answer token, while inference TTFT starts at target-model inference.Source: B Detailed Experimental Setup · pp. 13–14

C.1 VTAT Breakdown

  • Target-model visual encoding takes 7.596 s on average, or 62.36% of VTAT, making it the largest component in the profiled encode-first pipeline.Inference TTFT accounts for 0.243 s (1.99%); the remaining 4.342 s (35.65%) is residual pipeline overhead.Source: Table 5 · p. 14, §C.1 · p. 14

C Additional Analysis of Empirical Observations

C.2 Task-Level Analysis of Fixed Evidence Policies

Fixed semantic and recent evidence policies favor different tasks, while the per-query oracle improves over both. Across online evaluations, WRWS shows task- and budget-dependent results, including comparisons with HERMES.

D.1 Detailed Results on the Main Online Benchmarks

  • On OVO-Bench, WRWS improves multiple tasks over SimpleStream across Qwen2.5-VL and Qwen3-VL models, with Qwen3.5 gains concentrated on all six Real-Time tasks at both evaluated scales.For LLaVA-OneVision, WRWS scores higher than reported HERMES on six of nine Backward and Real-Time tasks at 7B and five at 0.5B.Source: Table 7 · p. 15, §D.1 · p. 15

C.2 Task-Level Analysis of Fixed Evidence Policies

  • The per-query Oracle beats both fixed policies on every task for two backbones, supporting query-level adaptation rather than one globally fixed choice.Recent-only is stronger on all six Real-Time tasks for Qwen3-VL-8B and Qwen2.5-VL-7B, while Semantic-only leads on selected other tasks.Source: §C.2 · p. 15

D.2 Comparison with HERMES under the 2-fps Setting

  • Under HERMES’s 2-fps setting, WRWS raises Qwen3-VL-8B and 4B Real-Time averages from 73.21 to 80.60 and 71.90 to 78.64, respectively.On StreamingBench, WRWS is lower at 8B (80.39 vs. 81.32) and higher at 4B (80.07 vs. 78.40); HERMES remains stronger on ASI and ATR at both scales.Source: Table 8 · p. 16, §D.2 · p. 16

D.3 Effect of the Target-Model Visual-Processing Budget

  • At K = 16, OVO-Bench overall reaches 65.41, while the Real-Time average falls from 84.94 at K = 6 to 81.29.The preferred budget varies by temporal evidence needs; scores remain close for K = 4, 6, and 8.Source: §D.3 · p. 16

D.4 Results across Model Scales

WRWS remains close to SimpleStream across evaluated model scales, while adaptive allocation outperforms fixed semantic–recency weights on two backbones. The relative utility of semantic and recent evidence varies with retrieval uncertainty.

E.1 Adaptive vs. Fixed Evidence Allocation

  • Adaptive allocation scores 55.98 on Qwen2.5-VL-7B and 58.59 on Qwen3-VL-8B, exceeding the best fixed-α scores of 55.73 and 58.25.The best fixed α differs by backbone, and preferred weights also vary across tasks.Source: §E.1 · p. 17

D.4 Results across Model Scales

  • Across Qwen3-VL scales, WRWS differs from SimpleStream by −1.06 to +0.67 points; at 30B-A3B it is higher by 0.67 points.At Qwen2.5-VL 72B, the difference is 0.07 points despite WRWS using K = 4 versus K = 16 for SimpleStream; overall performance remains comparable without model-specific K adjustment within each family.Source: §D.4 · p. 17, Table 12 · p. 18

E.2 Uncertainty-Conditioned Utility of Semantic and Recent Evidence

  • As uncertainty increases, Recent-only gains a larger advantage over Semantic-only, with gaps of −9.59 and −7.82 points in the two highest-uncertainty quintiles.The passage describes uncertainty as associated with relative downstream utility; semantic retrieval remains more competitive at low uncertainty.Source: §E.2 · p. 18

F Additional Efficiency Analysis

WRWS reduces vision-to-answer time relative to SimpleStream while keeping target-model visual context bounded. Qualitative cases show adaptive allocation can retain historical evidence alongside recent context, but selection and downstream reasoning errors remain, and retrieval-cache growth constrains long-stream scalability.

F Additional Efficiency Analysis

  • 5.84 s average VTAT for WRWS versus 12.18 s for SimpleStream across 3,035 OVO-Bench samples, or 47.93% of SimpleStream latency.VTAT ratios were 40.92%, 43.44%, and 57.74% for Backward, Real-Time, and Forward tasks, respectively.Source: F Additional Efficiency Analysis · p. 18, Table 17 · p. 21
  • Inference TTFT was 0.2431 s for WRWS and 0.2429 s for SimpleStream, with similarly small differences across task groups.Both methods use the same target-model visual encoding and answer generation after the query arrives; the sub-millisecond differences are likely normal measurement noise.Source: F Additional Efficiency Analysis · p. 18, Table 17 · p. 21
  • WRWS used 20.76 GB of GPU memory versus 17.28 GB for SimpleStream, while forwarding only selected K observations to the target model.The extra memory mainly comes from SigLIP and cached historical retrieval representations; offloading the cache to CPU maintains a length-stable GPU-memory profile similar to SimpleStream.Source: F Additional Efficiency Analysis · pp. 18–19

G Qualitative Analysis and Failure Cases

  • In successful examples, WRWS retains relevant earlier observations alongside recent context when needed, and concentrates allocation near query time when recent observations suffice.These examples illustrate adaptive semantic–recency allocation rather than a fixed temporal policy.Source: G Qualitative Analysis and Failure Cases · p. 19, Figure 6 · p. 21
  • Failures arise both when the target Video-LLM errs despite relevant evidence being selected and when recency-heavy selection omits needed historical evidence.The examples show that uncertainty-adaptive allocation does not eliminate either evidence-selection errors or downstream visual recognition and reasoning errors.Source: G Qualitative Analysis and Failure Cases · p. 19, Figure 7 · p. 21

H Limitations and Future Work

  • The retrieval cache grows with stream length, while cache compression or eviction and improved uncertainty calibration or richer retrieval signals remain future-work directions.The cached retrieval representations are more compact than target-model visual states and can be offloaded from GPU memory.Source: H Limitations and Future Work · p. 20

Related papers

Loading 2609.37345v1…