PaperlayerView exact revision on arXiv

Source-linked AI summary

CoFiE: Coarse-to-Fine Evidence Selection for Efficient Streaming Video Understanding

Jing Jiang, Yiran Ling, Ruonan Li, Dimitrios Stamoulis, Jie Liu

arXiv:2609.03675v2cs.CV

TL;DR

Streaming video models must answer questions over growing visual streams, while existing reduction methods mainly prune after frame encoding. CoFiE filters visually novel frames before encoding and refines the retained evidence for the query during prefill. It reports state-of-the-art accuracy–efficiency trade-offs and outperforms previously best methods by up to 3.15%.

  • Problem

    Streaming VLLMs must answer timestamped questions over growing video streams, but existing reduction methods prune after encoding and can miss rare answer-critical events.

    Source: §1 · p. 1

  • Method

    CoFiE first uses query-agnostic visual novelty to retain candidate frames before encoding, then applies query-specific evidence refinement during multimodal prefill.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    Up to 3.15%: CoFiE outperforms previously best methods across state-of-the-art streaming video understanding benchmarks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    CoFiE achieves state-of-the-art accuracy–efficiency trade-offs on streaming and long-video benchmarks, supporting frame-level evidence selection as a path toward low-latency streaming VLLMs.

    Source: §6 · p. 9

    CoFiE does not jointly schedule, compress, retrieve, or evict KV-cache states, which may still dominate memory as streams lengthen or multi-turn interactions accumulate.

    Source: §5 · p. 8

Abstract

from arXiv · show

Streaming video understanding requires Vision Language Models (VLLMs) to process growing video streams and answer user questions under tight latency constraints. Existing methods improve efficiency through token pruning and memory-bank schemes, but mainly reduce visual tokens after visual encoding. Consequently, downstream token pruning alone cannot substantially reduce end-to-end latency because the expensive frame encoding cost has already been incurred. We propose CoFiE, a Coarse-to-Fine Evidence Selection framework that decouples evidence selection into a coarse, query-agnostic filtering stage before the vision encoder and a fine, query-specific refinement stage during LLM prefill. CoFiE introduces Novelty-Guided Frame Filtering to retain visually distinctive candidate frames and Query-Specific Evidence Refinement to select the frames most relevant to the user query. This design removes substantial redundancy before frame encoding while preserving query-specific refinement once semantic information becomes available. Experiments show that CoFiE establishes a new state-of-the-art accuracy-efficiency trade-off across multiple video understanding benchmarks, reaching 78.86% accuracy on StreamingBench and 68.72% on OvO-Bench, with improvements of up to 3.15% over prior methods. Even with up to 80% evidence-frame filtering, CoFiE outperforms strong open-source multimodal models while improving end-to-end inference latency by up to 2.54 times.

1 Introduction

Streaming VLLMs must answer timestamped questions over growing video streams, but post-encoding pruning still incurs the expensive frame-encoding cost. CoFiE separates early, query-agnostic filtering from later, query-specific refinement and reports better benchmark performance than prior best methods.

1 Introduction

  • Up to 3.15%: CoFiE outperforms previously best methods across state-of-the-art streaming video understanding benchmarks.The introduction presents this as the outcome of its comprehensive evaluation.Source: Figure 1 · p. 1, §1 · p. 2
  • Streaming models must answer timestamped queries over a growing stream, so query-specific evidence is identified only after the question and visual tokens are processed together.Early reduction can discard short-lived events that matter to later questions.Source: §1 · p. 1
  • Existing approaches prune mainly after frame encoding, leaving encoding computation fully paid, while fixed-rate rules may miss rare answer-critical events.The resulting end-to-end gains remain limited relative to unmodified VLLMs processing denser visual context.Source: §1 · pp. 1–2
  • CoFiE uses pre-encoder histogram-based novelty scoring to retain visually distinctive, high-recall candidates, then ranks them by question relevance using text-to-visual attention during LLM prefill.The two lightweight stages separate query-agnostic coarse selection from query-specific refinement.Source: §1 · p. 2
  • CoFiE is presented as the first framework to perform pre-encoder frame-level pruning for streaming VLLM inference.The paper also reports state-of-the-art performance across multiple streaming and long-video benchmarks.Source: §1 · p. 2

2 Related Work

Prior streaming methods reduce visual context through sampling, memory, compression, or token pruning, but fixed compression ratios and single-stage rules can be brittle when evidence depends on both temporal change and the query. CoFiE distinguishes itself by separating early visual reduction from later query-specific refinement.

2 Related Work

  • Streaming VLLMs face substantial inference costs as visual context accumulates and must remain responsive while answering under partial observation.They also need to preserve useful history as new frames arrive.Source: §2 · p. 2
  • Prior approaches maintain compact video representations with memory banks, remove tokens based on temporal change, or use dynamic or question-aware pruning.Examples include persistent event banks, memory-augmented retrieval, and disentangled perception and decision events.Source: §2 · p. 2
  • Fixed compression ratios and single-stage reduction can be brittle because relevant evidence depends on both temporal change and the user query.CoFiE separates early visual reduction from answer-specific selection once the query is available.Source: §2 · pp. 2–3

3 Methodology

CoFiE first filters frames before visual encoding using query-agnostic signals, then refines the encoded candidates with query-specific evidence during prefill.

3.1 Problem Formulation & CoFiE Overview

  • CoFiE keeps a compact set of potentially evidential frames before encoding, then ranks encoded candidates with the question to select frames useful for answer generation.The first stage is query-agnostic; the second uses text-to-visual attention during LLM prefill.Source: §3.1 · p. 3

3.2 Novelty-Guided Frame Filtering

  • NGFF scores sampled frames with grayscale-histogram change against both the immediately previous frame and latest candidate, capturing abrupt changes and accumulated drift.Taking the maximum of the two distances allows either kind of change to trigger retention.Source: §3.2 · p. 4
  • NGFF always retains endpoints, adds uniformly spaced frames below Mmin, and retains frames when novelty exceeds threshold or candidate gaps reach Lmax.Lmax is measured in sampled-frame indices; only the resulting candidate frames are sent to the encoder.Source: §3.2 · p. 4

3.3 Query-Specific Evidence Refinement

  • During prefill, QER uses last-layer text-to-vision attention; each visual token takes its strongest textual association before averaging across that frame.This max-over-text aggregation is intended to preserve frames relevant through a small visual region or a single queried object.Source: §3.3 · pp. 4–5
  • QER retains top-scoring candidates at keep ratio ρ, removes other frames’ hidden states and compatible cached states, then continues generation on the shortened sequence.Scores are computed in chunks to avoid materializing the full text-by-vision attention matrix.Source: §3.3 · p. 5

4 Experiments

Experiments evaluate CoFiE on streaming and offline video benchmarks, measuring accuracy–efficiency trade-offs and testing its selection stages. Results show strong online performance, offline gains, and complementary benefits from the two filtering stages.

4.1 Experimental Setup

  • Evaluation spans StreamingBench and OvO-Bench streaming benchmarks plus four offline benchmarks, reporting benchmark metrics and inference time; total drop counts sequential NGFF and QER removals.Source: §4.1 · p. 5

4.2 Main Results: Online Benchmarks

  • 78.86% StreamingBench accuracy at 50% drop beats StreamForest by 1.60%; 68.72% OvO-Bench average score at 50% drop beats Qwen3-VL-8B-Instruct by 3.69%.Source: §4.2 · p. 5, Table 1 · p. 6, Table 2 · p. 7
  • Across drop ratios up to 90%, CoFiE’s fine-tuned checkpoint achieves higher OvO-Bench accuracy than the vanilla Qwen3-VL-8B checkpoint under identical filters.Source: Figure 3 · p. 6
  • At 50% and 80% drop, CoFiE delivers 1.49× and 2.54× end-to-end speedups, versus TimeChat-Online’s 1.16× and 1.18×.The comparison uses the same backbone; CoFiE’s gain mainly comes from reducing computation before visual encoding.Source: Figure 4 · p. 6, §4.2 · p. 7

4.4 Ablation Studies

  • QER raises NGFF20% scores at 50% total drop and improves NGFF50% and NGFF60% at 80% and 90% total drop; the 80% variant retains 99.9% of no-drop backbone accuracy.Source: Table 4 · p. 8, §4.4 · p. 8

4.3 Offline Long-video Results

  • Against strongest reported online baselines, CoFiE gains 7.79% on MLVU and 10.86% on LongVideoBench; it also improves unpruned Qwen across reported offline benchmarks.Source: §4.3 · pp. 7–8, Table 3 · p. 7

4.4 Ablation Studies

  • At 20% drop, Histogram Difference reaches 63.58% accuracy and 8.2% end-to-end speedup while retaining 96.9% of baseline accuracy.It has the best speed–accuracy trade-off among the tested practical front-end signals.Source: §4.4 · p. 8, Table 5 · p. 8

5 Limitations and Future Work

CoFiE reduces visual-token load through frame-level evidence selection but does not explicitly manage the KV cache. Historical KV states may still dominate memory as streams lengthen or multi-turn interactions accumulate.

5 Limitations and Future Work

  • CoFiE does not explicitly optimize KV-cache management.The framework focuses on frame-level evidence selection.Source: §5 · p. 8
  • Historical KV states may still dominate memory as video streams lengthen or multi-turn interactions accumulate.The paper identifies this as a potential bottleneck.Source: §5 · p. 8
  • Future work aims to combine frame selection with KV-cache management to better control memory growth while preserving query-relevant visual evidence.Source: §5 · p. 8

6 Conclusion

CoFiE first uses query-agnostic visual novelty to retain candidate frames before encoding, then refines them for the query during multimodal prefill. Across streaming and long-video benchmarks, it achieves state-of-the-art accuracy–efficiency trade-offs.

6 Conclusion

  • CoFiE achieves state-of-the-art accuracy–efficiency trade-offs on streaming and long-video benchmarks.Source: §6 · p. 9
  • CoFiE retains a high-recall set of candidate frames using query-agnostic visual novelty before visual encoding.Source: §6 · p. 9
  • During multimodal prefill, query-specific refinement selects evidence frames for generation, reducing redundant visual computation while preserving question-relevant evidence.Source: §6 · p. 9

A Implementation Details

CoFiE is implemented on Qwen3-VL-8B-Instruct, with NGFF filtering frames before visual encoding and QER refining candidates during LLM prefill. Fine-tuning and evaluation use a frozen ViT encoder and a 30%–50% frame retention ratio.

A Implementation Details

  • CoFiE is implemented on Qwen3-VL-8B-Instruct, with the ViT encoder frozen while the multimodal projector and LLM are fine-tuned.Source: A Implementation Details · p. 11
  • NGFF uses 32-bin grayscale histograms to remove temporally redundant frames before the computationally intensive vision encoder.Source: A Implementation Details · p. 11
  • QER scores query–frame relevance from the last-layer text-to-vision attention map during LLM prefill; final frame retention is set to 30%–50%.Source: A Implementation Details · p. 11

B.1 Frame Dropping versus Token Dropping

On OvO-Bench, frame-level dropping outperforms token-level dropping in end-to-end speed and retained performance. NGFF performance varies by less than 1% across tested Mmin and Lmax ranges, while τh controls the retention–latency trade-off.

B.1 Frame Dropping versus Token Dropping

  • 2.54× speedup with 101.5% of no-drop performance at 80% drop: CoFiE exceeds token-level dropping’s 1.18× speedup and retains 87.8% of unpruned accuracy.The comparison is on OvO-Bench Real-Time Visual Perception, with speedup relative to the unpruned Qwen3-VL-8B baseline.Source: §B.1 · p. 11, Table 6 · p. 11
  • At 50% drop, CoFiE takes 1.81s and reaches 68.72 accuracy, versus TimeChat-Online’s 2.31s and 60.48 accuracy.Source: §B.1 · p. 11, Table 6 · p. 11
  • Frame-level dropping accelerates end-to-end inference by removing redundant visual computation before the vision encoder, unlike token-level dropping after that cost is paid.Source: §B.1 · p. 12

B.2 NGFF Hyperparameter Sensitivity

  • NGFF performance varies by less than 1% across tested Mmin and Lmax ranges, while τh directly controls candidate-frame count and the retention–latency trade-off.Mmin is a lower bound active for very short videos with frequent visual changes; Lmax prevents long static intervals from becoming overly sparse.Source: §B.2 · p. 12

B.3 Allocation between NGFF and QER

The NGFF–QER allocation sets different accuracy–efficiency operating points, while histogram-difference filtering outperforms CLIP-based filtering on accuracy and latency. The results indicate that the stages offer complementary capabilities and that extra feature-extraction cost can offset filtering gains.

B.3 Allocation between NGFF and QER

  • The 50% NGFF / 60% QER split offers a favorable efficiency–accuracy balance, while 20% NGFF / 75% QER favors accuracy, indicating complementary stages.The allocation study used Qwen3-VL-8B on a Video-MME subset.Source: §B.3 · p. 12, Table 8 · p. 12

B.4 Comparison with Model-Based Filtering

  • Histogram-difference filtering achieves higher accuracy and lower latency than CLIP-based filtering, whose feature-extraction cost largely offsets its latency reduction.The comparison used CLIP ViT-B/32 frame features on an OvO-Bench subset.Source: §B.4 · p. 12, Table 9 · p. 12

B.5 Attention-Layer Selection for QER

Middle-to-late attention layers outperform early layers for QER on the evaluated StreamingBench subset. Layer 24 achieves the highest accuracy, while the method uses the final layer for its output association and cross-architecture generalization.

B.5 Attention-Layer Selection for QER

  • The method uses the final layer because its attention signal is directly associated with output prediction and generalizes without layer-index tuning; Layers 18–35 are robust choices.The reported strong results across Layers 18–35 support robustness to layer choice within that range.Source: §B.5 · pp. 12–13

B.6 Preservation of Answer-Relevant Evidence

NGFF retains more answer-relevant evidence than uniform or random sampling at both tested drop ratios, with a wider gap at 50% drop. CoFiE also transfers across the two evaluated VLLM families, improving accuracy while providing speedups at a 50% total drop.

B.6 Preservation of Answer-Relevant Evidence

  • Answer-relevant retention is assessed using full-video QER attention from the question and four answer options, via Attention Preservation Ratio and Top-Evidence Recall@20%.Experiments use Qwen3-VL-8B on a StreamingBench subset.Source: §B.6 · p. 13
  • At both tested drop ratios, NGFF retains more answer-relevant attention and more top-evidence frames than uniform or random sampling; the gap widens at 50% drop.The reported pattern is consistent with NGFF’s role as a coarse candidate selector before query-specific refinement.Source: §B.6 · p. 13

B.7 Generalization across VLLM Backbones

  • At a 50% total drop, Qwen3-VL-8B accuracy improves by 0.80 percentage points with 1.65× speedup; InternVL3.5-8B accuracy improves by 2.40 points with 1.15× speedup.The cross-backbone results are on a StreamingBench subset and show transfer across the two VLLM families.Source: §B.7 · p. 13, Table 12 · p. 13
Loading 2609.03675v2…