Source-linked AI summary
OASIS: On-Demand Hierarchical Event Memory for Streaming Video Reasoning
Zhijia Liang, Jiaming Li, Weikai Chen, Yanhao Zhang, Haonan Lu, Guanbin Li
TL;DR
Streaming video reasoning must find sparse decisive evidence in history that grows without bound. OASIS uses hierarchical event memory with short-context reasoning first and semantic retrieval only when needed. Across backbones and benchmarks, it reports stronger long-horizon and compositional reasoning with bounded token cost, while remaining training-free.
Problem
Streaming video reasoning must handle unbounded temporal history containing sparse query-relevant evidence without drowning attention or erasing decisive details through compression.
Method
OASIS maintains hierarchical event memory and performs short-context reasoning first, invoking semantically guided historical retrieval when accessible evidence is insufficient.
Results
OASIS improves long-horizon correctness and compositional reasoning across multiple MLLM backbones while reducing active window budgets; Qwen3-VL-8B gains +11.35 in OVO-Bench Perception.
Takeaways & Limitations
Structured, on-demand access to temporal history provides a training-free way to improve streaming reasoning while keeping token and memory demands bounded.
Takeaways & Limitations
The training-free zero-shot regime remains limited on tasks requiring multi-hop retrieval or fine-grained visual grounding.
Abstract
from arXiv · showhide
Streaming video reasoning requires models to operate in a setting where history grows without bound while meaningful evidence remains scarce. In such a landscape, relevant signal is like an oasis-small, critical, and easily lost in a desert of redundancy. Enlarging memory only widens the desert; aggressive compression dries up the oasis. The real difficulty lies in discovering where to look, not how much to remember. We therefore introduce OASIS, a novel framework for streaming video reasoning that tackles this challenge through structured, on-demand retrieval. It organizes streaming history into hierarchical events and performs reasoning as controlled refinement-short-context inference first, followed by semantically grounded retrieval only when uncertainty arises. As the retrieval is driven by high-level intent rather than embedding similarity, the retrieved memory is substantially more accurate and less noisy. Additionally, the mechanism is plug-and-play, training-free, and readily attaches to different streaming MLLM backbones. Experiments across multiple benchmarks and backbones show that OASIS achieves strong gains in long-horizon accuracy and compositional reasoning with bounded token cost and low request delay. Code is available at https://github.com/Solus-sano/OASIS.
1. Introduction
Streaming video reasoning must identify sparse, decisive evidence within an unbounded and redundant history. OASIS addresses this as temporal routing through hierarchical memory and short-context reasoning with on-demand semantic retrieval.
- Motivation: Streaming video reasoning is difficult because temporal context grows without bound while query-relevant information remains sparse.Retaining all frames overwhelms attention, whereas permanent compression can erase decisive evidence.
- Motivation: The central challenge is locating the decisive historical region rather than increasing storage, compression, or token capacity.The paper frames streaming video reasoning as a temporal routing problem.
- OASIS: OASIS organizes history into a high-fidelity short window, a medium-resolution buffer, and a multi-resolution event hierarchy updated online.The hierarchy is built through segmentation and structural merging to preserve semantically navigable long-term memory.
- Two-phase reasoning: OASIS first reasons over recent context and summaries, then semantically routes retrieval to historical evidence only when the short context is insufficient.This avoids inflating the context window and differs from retrieval based directly on raw question similarity.
- Contributions: OASIS is plug-and-play and training-free, attaches to different streaming MLLM backbones, and reports gains in long-horizon and compositional reasoning with bounded token cost.The introduction characterizes these improvements across multiple backbones and benchmarks.
- Related work: Prior video methods use sparse sampling, pooling, or retrieval, but fixed-video assumptions and rigid retrieval can omit temporal evidence or require reprocessing as streams grow.OASIS is positioned for continuously growing streams rather than fully observed prerecorded videos.
3. Method
OASIS combines multi-resolution event memory with a two-phase reasoning policy for streaming video. It maintains bounded, semantically navigable history and retrieves distant evidence only when coarse reasoning is insufficient.
- Hierarchical Event Memory: OASIS maintains a high-fidelity short window, medium-resolution buffer, multi-resolution event hierarchy, and QA summary for different temporal resolutions.
- Hierarchical Event Memory: The short window preserves fine-grained present details, while the medium buffer covers a wider recent range at a lower frame rate.
- Hierarchical Event Memory: The Event Forest processes sequential non-overlapping windows into key-frame-based event nodes, keeping its representation synchronized with the live stream.
- Hierarchical Event Memory: When roots exceed a threshold, OASIS greedily merges the temporally adjacent pair with the highest hierarchy-aware score to bound memory growth.
- Two-phase Reasoning: Coarse reasoning uses immediately accessible memory first; if information is insufficient, fine reasoning generates retrieval cues and searches the event hierarchy.
- Two-phase Reasoning: The policy avoids prohibitive token overhead and attention collapse by limiting root summaries, while strategic retrieval supports accurate temporally consistent understanding.
4. Experiment
OASIS is evaluated across streaming benchmarks, backbones, ablations, token consumption, and latency. Results show gains in present-scene perception and backward retrieval, while ablations support its hierarchical memory and reasoning policy.
- Evaluation setup: OASIS is evaluated on OVO-Bench, StreamingBench, and StreamBench using Qwen3-VL-8B, Qwen2.5-VL-7B, and GLM-4.6V.The comparisons include offline and online state-of-the-art models under online inference protocols.
- Main results: +11.35 Perception and +6.02 Backward gains are reported for Qwen3-VL-8B on OVO-Bench with OASIS.The Perception result targets present-moment grounding, while the Backward result reflects long-term retrieval.
- Main results: OASIS produces consistent gains on StreamBench and generalizes across multiple streaming MLLM backbones.The supplied passages state consistent StreamBench gains and report cross-backbone evaluation on OVO-Bench.
- Ablations: 77.66 Perception and 51.93 Backward are obtained with only the short window, while adding the Event Forest raises Backward to 57.21.Using only the medium buffer lowers Perception to 74.31 but raises Backward to 54.00, illustrating different memory-resolution trade-offs.
- Ablations: 56.13 Backward accuracy with native RAG is below 57.21 with OASIS’s fine reasoning retrieval.The ablation directly retrieves with the original question, whereas OASIS generates retrieval cues conditioned on historical summaries.
- Efficiency: OASIS overlaps 7.73s node generation and 6.52s root merging with incoming video in a 32s buffer window.The reported latency measurements are part of the system’s asynchronous streaming implementation.
5. Conclusion
The paper frames streaming video reasoning as temporal routing: locating decisive evidence within overwhelming history. OASIS implements this through selective refinement with a training-free, model-agnostic hierarchical memory.
- Conclusion: OASIS treats streaming video reasoning as temporal routing over decisive evidence hidden within long temporal history.The conclusion emphasizes locating relevant evidence rather than retaining all history.
- Conclusion: OASIS reasons in short context by default and retrieves long-term evidence only when semantically required.This selective-refinement policy is presented as the framework’s central operating principle.
- Conclusion: OASIS integrates with existing MLLMs without training while improving efficiency and accuracy.The conclusion describes the approach as model-agnostic and training-free.
6. Benchmarks
The evaluation uses benchmarks designed for temporally constrained, interactive streaming video understanding. They cover immediate perception, contextual reasoning, long-term memory, and hour-scale temporal reasoning.
- OVO-Bench: OVO-Bench tests online video understanding when answers must use only information available up to the playback time.It contains 644 videos across seven domains and includes Perception and Backward Tracing scenarios.
- StreamingBench: StreamingBench evaluates multi-turn streaming interaction across 900 videos and 4,500 questions.Its axes include Real-Time Visual Understanding, MCU, ACU, SQA, and Proactive Output.
- Long-horizon benchmark: HourVideo contains egocentric videos spanning 20–120 minutes, directly testing hour-scale temporal reasoning.The benchmark is used to evaluate performance beyond shorter and medium-duration settings.
- StreamBench: StreamBench focuses on open-ended, multi-turn online interaction across 306 videos and more than 1.8k questions.Its task categories include Object Search, Long-term Memory Search, Short-term Memory Search, and Conversational Interaction.
7. Long-Horizon Evaluation
Long-horizon evaluations show that OASIS remains beneficial on hour-long videos and becomes especially advantageous on clips exceeding 15 minutes. The largest reported gains occur in both forward/perception and backward subsets.
- HourVideo: OASIS improves Qwen3-VL-8B from 35.11% to 37.35% on HourVideo.HourVideo uses egocentric videos spanning 20–120 minutes.
- Long OVO-Bench clips: On videos longer than 15 minutes, OASIS raises forward/perception accuracy from 56.76% to 70.27%.This long-duration slice isolates the regime where historical context is substantially larger.
- Long OVO-Bench clips: On videos longer than 15 minutes, OASIS raises backward accuracy from 30.00% to 50.00%.The reported backward gain is larger than the average gains in the main paper.
- Interpretation: The long-duration results suggest that OASIS becomes increasingly advantageous as historical context grows and naive accumulation becomes more brittle.The authors connect this regime to the importance of long-term memory maintenance and retrieval quality.
8. Detailed Results on StreamBench
OASIS generalizes beyond tightly structured benchmarks, improving open-ended retrieval and reasoning across diverse StreamBench tasks. Its gains cover recent grounding, historical search, and multi-turn conversational consistency.
- OASIS achieves strong gains across StreamBench’s six diverse open-ended task categories.StreamBench uses open-ended questions and multi-turn interaction to test flexible retrieval and reasoning.
- Improvements on Long-term Memory Search and Short-term Memory Search benefit both historical retrieval and recent grounding.
- Gains on Conversational Interaction suggest that QA summaries and retrieval help maintain consistency across multi-turn dialogues.
9. Details of Retrieval Algorithm in Fine Reasoning
Fine reasoning retrieves diverse, nonredundant evidence from the Event Forest by pruning related nodes after selecting the highest-scoring candidate. This balances information diversity with event-appropriate granularity.
- Greedy pruning removes each selected node’s ancestors and descendants from the candidate pool.The procedure repeatedly selects the highest-scoring node until the retrieval budget is reached.
- The pruning strategy prevents simultaneous retrieval of parent and child nodes, forcing evidence selection from distinct event branches.
- The final selection maximizes information diversity while choosing the optimal granularity for each event.
10. Analysis of hyper-parameters
Sensitivity analyses show that OASIS is robust to λ, while retrieval-count choices involve a performance–efficiency trade-off. Sequential question-answering accuracy improves with more question history within the tested range.
- Sensitivity Analysis of λ: Less than 1% separates OVO-Bench Perception Avg across the tested λ values.λ controls the trade-off between event-node similarity and hierarchical level.
- Sensitivity Analysis of kf: Increasing kf from 1 to 2 significantly improves performance, whereas kf = 3 yields marginal Perception gains and slightly lower Backward performance.The authors select kf = 2 to balance performance and computational efficiency.
- Sensitivity Analysis of kq: SQA accuracy increases with kq across the tested range, reaching 49.20 at kq = 3.kq controls the number of previous questions retrieved from question memory.
11. Computational Efficiency Analysis
OASIS reduces the memory burden of streaming video reasoning by replacing full-history processing with bounded hierarchical memory and selective retrieval. Its ablations and algorithmic design target efficient, nonredundant evidence access.
- Peak GPU Memory Footprint: 76.59 GB is consumed by the baseline, largely because its KV-cache grows linearly with the entire video history.This footprint approaches the limit of a single A800 80GB GPU.
- Retrieval Algorithm: The Greedy Pruning Strategy selects high-scoring event nodes while removing their ancestors and descendants to avoid redundant lineage retrieval.
- Peak GPU Memory Footprint: The Event Forest adds minimal overhead, and OASIS fits within consumer-grade hardware limits or supports larger enterprise-GPU batch sizes.
- Peak GPU Memory Footprint: 28.48 GB is consumed by the full OASIS framework, substantially reducing memory requirements relative to the baseline.
- Peak GPU Memory Footprint: OASIS’s memory design combines an Event Forest, Medium Buffer, and Short Window, whose component-wise effects are evaluated against the full-context baseline.
12. Qualitative Analysis
OASIS is qualitatively evaluated on real-time perception and long-term retrieval queries. The examples show that coarse reasoning preserves present-focused perception, while fine reasoning recovers evidence missing from the short context.
- OASIS is illustrated on real-time perception and long-term retrieval queries in streaming scenarios.The examples are selected to represent these two query types.
- Real-time Perception: In the real-time perception example, full-context modeling misattends to historical cards despite the cue “now.”The resulting answer is incorrect because attention collapses onto historically presented cards.
- Real-time Perception: OASIS’s coarse reasoning correctly identifies the five cards currently on the table from the Short Window.This demonstrates the benefit of grounding perception in the immediate context.
- Long-term Retrieval: The coarse-reasoning-only variant fails on a query requiring evidence absent from the ShortWindow and summaries.It incorrectly concludes that the white plate never appeared.
- Long-term Retrieval: The full OASIS framework bridges missing evidence by identifying the information deficiency and using retrieval.The example contrasts full OASIS with the coarse-reasoning-only variant.
13. Details of Prompts
The implementation details specify prompts for two-stage reasoning and hierarchical event-memory maintenance. The section also records a current limitation: training-free zero-shot reasoning remains challenged by multi-hop retrieval and fine-grained visual grounding.
- 13. Details of Prompts: Figure 7 specifies the main inference model’s instructions for the Two-Stage Reasoning policy.
- 13. Details of Prompts: Figure 8 presents the prompt that generates event nodes from the medium buffer.
- 13. Details of Prompts: Figure 9 depicts instructions for dynamically merging adjacent event nodes into a unified summary.
- 13. Details of Prompts: Figure 10 outlines a prompt used for iteratively updating the system’s memory-related state.
- 14. Future Work: OASIS currently relies on training-free MLLM zero-shot reasoning.The limitation is stated as part of the future-work discussion.
- 14. Future Work: This regime is limited on tasks requiring multi-hop retrieval or fine-grained visual grounding.The proposed future direction is instruction tuning for temporal planning and learned search strategies.