PaperlayerView exact revision on arXiv

Source-linked AI summary

EGSD: Event-Grounded Self-Distillation for Streaming Video Understanding

Yuwei Miao, Xuesheng Zhang, Wenhao Zou, Jixia Zhang, Jianwei Lv, Bo Yuan, Junfeng Wang, Shiao Xie

arXiv:2609.36803v3cs.CV

TL;DR

Streaming video requires memory written before future questions arrive, but standard OPSD’s question-aware teacher can misalign with the student and encourage memory collapse. EGSD combines multiplicative gating with Event-based teacher re-weighting and entity-coverage reward, and reports stronger memory diversity and answering accuracy across online and offline benchmarks.

  • Problem

    Streaming memory must be written before the question arrives, while standard OPSD scores it using question-and-GT privilege, creating an end-to-end optimization mismatch and favoring question-relevant entities over diverse memory.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    EGSD treats streaming memory as updates over verifiable Events and combines multiplicative teacher gating, Event-privileged re-weighting, and an entity-coverage reward.

    Source: §1 · p. 2

  • Results

    EGSD improves memory diversity and answering accuracy across online and offline benchmarks; effective-entity recall rises 17.4% with 6.8% more memory length.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The paper concludes that Event-grounded self-distillation improves memory diversity and answering accuracy for streaming video understanding.

    Source: §5 · p. 10

Abstract

from arXiv · show

Real-time video understanding requires incrementally maintaining a memory of streaming content, and optimizing this requires dense process signals. On-Policy Self-Distillation (OPSD), which lets one model serve as both teacher and student with the teacher receiving additional privileged information such as the question and ground-truth (GT) answer, can supply such token-level signals. However, applying it directly to streaming video raises two problems. (1) The student cannot be optimized end-to-end, where memory is written before the question arrives, yet the teacher scores it with the question-and-GT privilege, misaligning their preferences. (2) Effective-entity memory collapses, where the question-and-GT privilege makes the teacher favor only question-relevant entities, and token-mean averaging over a memory renders its signal invariant to how many entities that memory covers, both driving memory against the streaming need for diversity. To address these issues, we propose Event-Grounded Self-Distillation (EGSD), which characterizes streaming memory as an incremental update over verifiable Events (key visual entities, actions, and details) and targets the two problems on this basis. For problem (1), we adapt the OPSD signal into a multiplicative weight combined with the outcome reward; for problem (2), we re-weight the teacher with Events as privileged information to counter its question-relevance bias, and add an entity-coverage reward to supply the coverage preference the token-mean teacher lacks. Extensive experiments on mainstream online and offline benchmarks show EGSD achieves strong performance, reaching 79.8% on StreamingBench and 73.4% on the OVO-Bench Real-Time track, while memory analysis shows effective-entity recall rises 17.4% at only 6.8% more memory length.

1 INTRODUCTION

Streaming video requires memory written before a future question arrives, but existing OPSD can misalign teacher and student preferences and collapse memory diversity. EGSD uses verifiable Events and complementary training signals to address these problems; its memory analysis reports higher effective-entity recall with a small increase in memory length.

  • Streaming memory is written before the question arrives, while the teacher sees the question and ground-truth answer, misaligning their preferences and obstructing end-to-end optimization.The student must retain diverse visual entities because low-latency answers rely on existing textual memory and the current clip.Source: §1 · pp. 1–2
  • Question-and-GT privilege favors question-relevant entities, while token-mean averaging is indifferent to how many effective entities memory covers, driving memory away from diversity.Figure 1 depicts memory diversity collapsing under OPSD/RLSD.Source: §1 · p. 2, Figure 1 · p. 2
  • EGSD treats memory as incremental updates over verifiable Events—entities, actions, and details—and combines multiplicative teacher gating, Event-privileged re-weighting, and an entity-coverage reward.Events are defined independently of the question and provide a stable grounding basis for the teacher and reward.Source: §1 · pp. 2–3
  • The paper analyzes and verifies the causes of end-to-end optimization failure and memory collapse under vanilla OPSD in streaming video.EGSD’s mechanisms are described as complementary: Event-based reward drives effective-entity writing, while multiplicative gating aligns optimization with the final answer.Source: §1 · p. 2
  • 17.4% higher effective-entity recall comes with only 6.8% more memory length, alongside strong performance on online and offline benchmarks.The reported result connects increased memory coverage with a modest increase in memory length.Source: §1 · p. 3

2 RELATED WORK

Prior work addresses long-video processing, process rewards, and on-policy self-distillation, but the cited approaches leave streaming-memory content without a trainable, reward-checkable objective. The paper positions Event-grounded memory as addressing that gap.

2.1 MULTIMODAL LLMS FOR LONG VIDEO AND STREAMING VIDEO REASONING

  • Long-video methods use context extension or token compression, while streaming memory may be visual tokens, hidden states, or natural language; none imposes a trainable, reward-checkable objective on memory content.Visual-token compression can irrecoverably drop content, hidden states are unreadable and unscorable, and natural-language memory is readable but costs tokens.Source: §2.1 · p. 3

2.2 VIDEO REINFORCEMENT LEARNING

  • Long-video process rewards evaluate where to look through coordinate-based grounding, zoom-in, temporal grounding, or joint sampler optimization, rather than what streaming memory should retain.The cited methods use interval annotations, while this paper applies its process reward to memory content.Source: §2.2 · p. 3

2.3 MULTIMODAL ON-POLICY SELF-DISTILLATION

  • Existing OPSD scores tokens produced toward a question the student already knows, whereas streaming memory is written before future queries arrive.The paper identifies this mismatch as the gap its work addresses.Source: §2.3 · p. 3

3 METHOD

EGSD treats streaming memory as an incremental update over verifiable Events and combines outcome-aligned teacher modulation with Event-based teacher signals and entity-coverage reward. Its method addresses the mismatch between question-blind memory writing and question-informed scoring, while encouraging memories to retain grounded entity diversity.

3.1 ANALYSIS OF THE OPSD SIGNAL IN THE STREAMING-VIDEO SETTING

  • The student writes memory before the question arrives, so question-informed teacher scores alone misalign with the student and cannot drive end-to-end optimization.EGSD lets the final result set update direction and the privileged teacher set credit magnitude across memory decisions.Source: §3.1 · p. 4

3.2 EGSD: EVENT-PRIVILEGED TEACHER RE-WEIGHTING AND ENTITY-COVERAGE REWARD

  • EGSD treats streaming memory as incremental updates over verifiable Events and combines Event-privileged teacher re-weighting with an entity-coverage reward.The teacher favors fact grounding, while the coverage reward ranks memories by how many effective entities they ground.Source: §3.2 · p. 4, §3.2.3 · pp. 5–6

3.2.1 EVENT: A TRIPLE-COMPONENT VERIFIABLE SIGNAL

  • Each clip’s Event fact set contains key entities, temporally ordered actions, and checkable details such as OCR, counts, and spatial relations.A frozen large model extracts the facts offline for use by the teacher and coverage reward.Source: §3.2.1 · p. 5

3.2.2 EVENT-BASED PRIVILEGED TEACHER

  • The privileged teacher conditions on the student state, Event facts, question, and ground-truth answer, extending its signal from question relevance to fact grounding.Its token weights modulate outcome-reward credit while preserving the outcome reward’s update direction; the teacher gate remains constant during training.Source: §3.2.2 · p. 5

3.2.3 EVENT-BASED ENTITY-COVERAGE REWARD

  • The entity-coverage reward ranks rollouts by grounded effective entities, supplying a coverage preference that token-mean teacher scoring lacks.Its coverage-biased harmonic signal weights coverage above grounding with βe = 1.5, while still penalizing ungrounded expansion.Source: §3.2.3 · pp. 5–6

4 EXPERIMENTS

Experiments compare EGSD with streaming and offline baselines across five video-understanding benchmarks. Results show gains on event-accumulation tasks and improved entity recall with limited additional memory length.

4.1 IMPLEMENTATION DETAILS / 4.2 BENCHMARKS AND BASELINES

  • Evaluation covers five online and offline benchmarks, with OPSD, RLSD, and EGSD trained from the same SFT checkpoint alongside open-source baselines.Experiments use Qwen2.5-VL-7B-Instruct and Qwen3-VL-8B-Instruct; memory writes occur every 5000 visual tokens under a shared streaming protocol.Source: §4.1 · p. 6, §4.2 · p. 7

4.3 MAIN RESULTS

  • EGSD beats VST by +10.2 on StreamingBench event understanding and by 2.8 on OVO-Bench Real-Time, while improving memory-dependent benchmarks.On StreamingBench, gains also cover temporal counting (+7.6), clip summarization (+1.9), and prospective reasoning (+1.9); on memory-dependent tests, gains include OVO-Bench Backward EPM and ASI (+4.0 each) and LongVideoBench (+2.9).Source: §4.3 · pp. 7–8, Table 2 · p. 7
  • On Qwen3-VL-8B, EGSD raises OVO-Bench Backward / Forward averages from 52.8 / 53.4 to 56.2 / 56.7; multiplicative RLSD improves most memory columns over Vanilla OPSD.Vanilla OPSD drops Qwen2.5-VL Backward from 49.0 to 46.0, while the teacher scores memory using question and ground-truth information unavailable when the student writes it.Source: §4.3 · p. 8

4.4 ABLATION STUDIES OF EGSD COMPONENTS

  • Table 3 ablates EGSD components one by one from Vanilla RLSD using the Qwen3-VL-8B backbone.The supplied passage identifies the ablation setup but gives no component-wise result values.Source: Table 3 · p. 8

4.5 MEMORY QUALITY ANALYSIS

  • Entity recall measures the fraction of reference Event entities grounded, while average memory character count tracks length overhead across four methods sharing an SFT start.The comparison includes Vanilla OPSD, Vanilla RLSD, EGSD, and EGSD without rmem.Source: §4.5 · p. 8, Figure 3 · p. 9

4.5.1 RECALL AND MEMORY-LENGTH ANALYSIS

  • 0.518 entity recall for EGSD is +17.4% over the SFT start, versus 0.400 (−9.4%) for Vanilla RLSD; EGSD uses 2303 characters, only +6.8%.Removing rmem lowers recall from 0.518 to 0.470 (−9.3%) and memory length from 2303 to 2124 characters.Source: §4.5.1 · p. 8

4.5.2 TOKENS REWARDED ANALYSE

  • EGSD’s visual-entity ratio among rewarded content words stays clearly above Vanilla RLSD throughout training, although the ratio is not a training target.The ratio counts rewarded content words matching clip-registered ENTITY words; the reported increase emerges after the teacher is re-weighted with Event facts.Source: §4.5.2 · p. 9, Table 4 · p. 9
  • On the same car-washing clip, EGSD rewards scene-grounded entities and actions, while Vanilla RLSD can penalize salient entities weakly tied to the question.EGSD gives high weight to the registered washing action and grounded objects, but its deepest penalty falls on the hallucinated brand Toyota.Source: §4.5.2 · pp. 9–10, Figure 4 · p. 9

5 CONCLUSION

EGSD treats streaming memory as incremental updates over verifiable Events and combines several mechanisms to address memory-optimization failures. The authors report gains in memory diversity and answering accuracy across online and offline benchmarks.

5 CONCLUSION

  • EGSD frames streaming video memory as incremental updates over verifiable Events and combines multiplicative teacher gating, Event-privileged re-weighting, and an entity-coverage reward.These mechanisms target the optimization failure and memory collapse of vanilla OPSD.Source: §5 · p. 10
  • Across online and offline benchmarks, EGSD improves memory diversity and answering accuracy for streaming video understanding.Source: §5 · p. 10
  • The authors state that generative AI tools were used only to polish author-written text, not for research or scientific claims and results.Authors reviewed all AI-assisted text and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A INFERENCE LATENCY

On the reported VideoHolmes comparisons, online streaming returns a first token sooner than offline with-CoT answers take to complete. The paper distinguishes TTFT from end-to-end latency because they measure different answering setups.

A INFERENCE LATENCY

  • TTFT measures question-to-first-token delay for direct-answer and online streaming methods, while E2E measures video-end-to-last-token time for offline with-CoT methods.Reporting TTFT for offline with-CoT would undercount its true cost by an order of magnitude.Source: A INFERENCE LATENCY · p. 13
  • On Qwen3-VL-8B, offline with-CoT takes 5.89 s E2E versus 0.73 s TTFT online; Qwen2.5-VL-7B takes 5.68 vs 0.90 s.The table reports VideoHolmes p50 latency; offline rows use E2E and online rows use TTFT.Source: Table 5 · p. 13, A INFERENCE LATENCY · p. 13
  • Streaming think completes asynchronously as clips arrive before the question, keeping first-token latency subsecond even with the added memory mechanism.On Qwen3-VL-8B, the sample-weighted median first token across seven benchmarks is 0.98 s, on par with the GT-only variant.Source: A INFERENCE LATENCY · p. 13
  • Because VST measurements use different hardware, the comparison supports orders-of-magnitude and online–offline gap claims, not absolute seconds.All authors’ rows use one harness and GPU batch with failure rate=0; other rows are quoted from the VST paper.Source: A INFERENCE LATENCY · p. 13

Related papers

Loading 2609.36803v3…