PaperlayerView exact revision on arXiv

Source-linked AI summary

CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Management

Chao Wang, Xudong Tan, Jianjian Cao, Kangcong Li, Tao Chen

arXiv:2603.19571v1cs.CV

TL;DR

Streaming video understanding faces unbounded visual-token growth and memory failures, while existing retention strategies may miss critical semantic transitions. CurveStream uses curvature-aware hierarchical memory management with real-time scoring and online K-Sigma thresholds, achieving substantial gains over streaming baselines and new state-of-the-art results.

  • Problem

    Streaming video produces linear visual-token growth that can cause OOM errors or catastrophic forgetting, while existing retention methods often lack intrinsic semantic awareness.

  • Method

    CurveStream is a training-free framework that uses feature-trajectory curvature, real-time Curvature Scores, and online K-Sigma thresholds to route frames into clear or fuzzy memory states.

  • Results

    CurveStream establishes new state-of-the-art streaming results and improves over specialized streaming methods by 6.84% on StreamingBench and 4.06% on OVO-Bench.

  • Takeaways & Limitations

    The lightweight, model-agnostic module provides a robust solution for continuous video perception with broad architectural compatibility.

Abstract

from arXiv · show

Multimodal Large Language Models have achieved significant success in offline video understanding, yet their application to streaming videos is severely limited by the linear explosion of visual tokens, which often leads to Out-of-Memory (OOM) errors or catastrophic forgetting. Existing visual retention and memory management methods typically rely on uniform sampling, low-level physical metrics, or passive cache eviction. However, these strategies often lack intrinsic semantic awareness, potentially disrupting contextual coherence and blurring transient yet critical semantic transitions. To address these limitations, we propose CurveStream, a training-free, curvature-aware hierarchical visual memory management framework. Our approach is motivated by the key observation that high-curvature regions along continuous feature trajectories closely align with critical global semantic transitions. Based on this geometric insight, CurveStream evaluates real-time semantic intensity via a Curvature Score and integrates an online K-Sigma dynamic threshold to adaptively route frames into clear and fuzzy memory states under a strict token budget. Evaluations across diverse temporal scales confirm that this lightweight framework, CurveStream, consistently yields absolute performance gains of over 10% (e.g., 10.69% on StreamingBench and 13.58% on OVOBench) over respective baselines, establishing new state-of-the-art results for streaming video perception.The code will be released at https://github.com/streamingvideos/CurveStream.

I. INTRODUCTION

Streaming video understanding is constrained by unbounded visual-token growth and memory failures, while existing retention methods often miss semantic transitions. CurveStream addresses this with curvature-aware, adaptive memory management and reports strong gains across streaming benchmarks.

  • Motivation: Streaming videos cause linear visual-token growth, creating OOM risks and catastrophic forgetting under fixed GPU memory constraints.The paper identifies continuous visual-memory management within a fixed budget as the central challenge for long-term streaming understanding.
  • Existing limitations: Uniform sampling, low-level motion metrics, rule-based compression, and query-driven retrieval can lack semantic awareness, disrupt coherence, or blur transient transitions.These limitations include semantic fragmentation, information blurring, and delayed real-time perception.
  • Key insight: High-curvature regions in feature trajectories align with critical global semantic transitions such as new events, viewpoint changes, and action boundaries.Curvature is presented as a geometric measure of semantic-shift intensity that is less dependent on local physical motion.
  • Method: CurveStream combines real-time Curvature Scores with a dynamic K-Sigma threshold to route frames into clear and fuzzy memory states under a fixed token capacity.The framework is training-free and designed for non-stationary streaming videos.
  • Evaluation: CurveStream is evaluated across 10 StreamingBench tasks, 6 OVOBench tasks, and offline datasets spanning 15–1200 seconds.Experiments cover LLaVA-OneVision and Qwen-VL model families from 4B to 32B parameters.
  • Results: 10.69% and 13.58% absolute gains are achieved on StreamingBench and OVOBench, respectively, using the Qwen2.5-VL-7B baseline.The reported accuracies are 84.00% on StreamingBench and 73.48% on OVOBench.

B. Existing Streaming Video Memory Management Mechanisms

Streaming video understanding requires online visual-memory scheduling because infinite streams cause unbounded token growth, while existing eviction and sampling strategies inadequately preserve semantically important information. CurveStream addresses this with curvature-aware scoring and hierarchical, fixed-capacity memory updates.

  • CurveStream framework: The framework combines a Curvature-Aware Scorer with Hierarchical Visual Memory Management to route incoming tokens through a fixed-capacity queue.HVMM uses Clear, Blurred, and Discard states, while CAS measures semantic transition intensity from feature trajectories.
  • Problem formulation: Infinite streaming sequences make feeding all historical frames computationally intractable, requiring a visual memory queue capped at Nmax.The formulation also accounts for MLLM parameter size and quadratic self-attention complexity.
  • Problem formulation: CurveStream formulates streaming understanding as online information extraction, producing retention and resolution decisions to update the memory bank.The update uses a state tuple (st, rt) and maximizes answer probability under |Mt| ≤ Nmax.
  • Curvature-Aware Scorer: The curvature-aware design targets failures of simple feature-difference sampling, which can overfit to large translational motions and local physical changes.The framework instead uses latent-manifold geometry to localize high-value information.
  • Curvature-Aware Scorer: CAS integrates first-order motion variation with second-order geometric curvature to characterize feature evolution and identify informative transitions.Curvature is approximated from angular deviation between adjacent feature-displacement vectors; abrupt directional changes increase the score.

C. Hierarchical Visual Memory Management (HVMM)

HVMM routes high-value frames into a fixed-capacity memory bank at differentiated resolution levels using temporally adaptive thresholds. This hierarchical routing is designed to suppress KV-cache growth while retaining important information.

  • Hierarchical Visual Memory Management: HVMM uses temporally adaptive dynamic thresholds to route high-value frames into a fixed-capacity memory bank at differentiated resolution levels.The mechanism is intended to suppress KV Cache bloat during streaming video processing.

1) Online Manifold Distribution Estimation:

HVMM estimates the evolving score distribution online and derives dual K-Sigma thresholds that adapt sensitivity to changing scene dynamics. These thresholds determine whether frames are retained clearly, compressed, or discarded under a fixed memory budget.

  • Online Manifold Distribution Estimation: HVMM treats high-value information filtering as an online distribution-aware process for adapting to real-time video pacing.Static thresholds can cause memory-bank collapse or loss of critical information in dynamically paced streams.
  • Online Manifold Distribution Estimation: The transient mean and variance of curvature scores are updated recursively with an Exponential Moving Average.The momentum factor γ controls the effective historical observation window.
  • Online Manifold Distribution Estimation: The method forms dual thresholds g1 = µt + k1σt and g2 = µt + k2σt, with k1 < k2, to scale sensitivity to scene intensity.The thresholds are derived from online distribution statistics rather than fixed in advance.
  • Hierarchical routing: Frames with CS_t ≥ g2 enter Clear Memory at high resolution because they indicate significant semantic shifts.The query-triggering frame is deterministically assigned to this state for immediate context awareness.
  • Hierarchical routing: Frames with g1 ≤ CS_t < g2 enter Blurred Memory at low resolution, while frames with CS_t < g1 are discarded as redundant.Low-resolution storage preserves necessary temporal associations while reducing token overhead.
  • Hierarchical routing: When memory exceeds capacity, strict FIFO eviction removes the oldest tokens regardless of retention state.This policy ensures a constant memory footprint and avoids OOM risks.

IV. EXPERIMENTS

CurveStream is evaluated against base MLLMs and specialized streaming methods under uniform token-budget constraints across streaming benchmarks. It delivers strong gains by using adaptive curvature-based memory allocation to preserve semantic transitions and reduce redundant visual content.

  • Evaluation Setup: Experiments compare base MLLMs and specialized streaming methods under a uniform maximum-token budget.The evaluation covers StreamingBench and OVOBench, with the adaptive memory queue dynamically changing between 10 and 20 frames.
  • Online Benchmark Results: 81.04% on StreamingBench and 70.73% on OVOBench are achieved with Qwen2-VL-7B, improving 12.00% and 10.08% over the listed uniform-sampling baseline.The corresponding baseline accuracies are 69.04% and 60.65%.
  • Online Benchmark Results: 84.00% on StreamingBench and 73.48% on OVOBench are achieved with Qwen2.5-VL-7B, yielding absolute gains of 10.69% and 13.58%.These results are reported for the Qwen2.5-VL-7B baseline with a dynamically sized 10–20-frame memory queue.
  • Online Benchmark Results: 85.56% on StreamingBench and 80.76% on OVOBench are achieved with Qwen3-VL-8B, improving 12.36% and 10.66% over its baseline.The listed Qwen3-VL-8B baseline scores are 73.20 and 70.1, respectively.
  • Online Benchmark Results: CurveStream improves accuracy by 6.84% on StreamingBench and 4.06% on OVOBench over recent specialized streaming methods, establishing a new training-free SOTA.The comparison includes methods such as FreshMem and HERMES.
  • Mechanism: Adaptive curvature-based memory allocation removes redundant static backgrounds and prioritizes high-frequency visual transition points for long-range reasoning.The paper attributes the leading performance to aligning the memory queue with the video’s dynamic evolution.

C. Offline Benchmark Results

CurveStream also generalizes beyond streaming video to offline short- and long-video benchmarks, while its gains remain stable across model parameter scales. The reported offline results include a specific VideoMME trade-off under constant memory usage.

  • Offline Benchmark Results: CurveStream shows strong generalization to conventional offline short- and long-video understanding tasks despite being designed for streaming scenarios.The offline evaluation includes MVBench and VideoMME.
  • Offline Benchmark Results: 1.03% on MVBench is gained with Qwen2.5-VL-7B, reaching 66.03% against the uniform-sampling baseline.MVBench is described as a fine-grained action-oriented short-video benchmark.
  • Offline Benchmark Results: 1.77% on VideoMME is gained with LLaVA-OneVision-7B, reaching 59.44%.VideoMME is described as a comprehensive long-video benchmark.
  • Offline Benchmark Results: Qwen2.5-VL-7B on VideoMME drops from 64.52% to 62.97% when constant memory usage is maintained over hours-long videos.The paper attributes this trade-off to preserving critical semantic transitions at the expense of some fine-grained global details.
  • Scalability Across Model Parameters: 8.7%, 12.4%, and 11.5% are gained on StreamingBench for Qwen3-VL models at 4B, 8B, and 32B scales, respectively.The corresponding OVOBench gains are 11.2%, 10.7%, and 10.6%.
  • Scalability Across Model Parameters: 11.2%, 10.7%, and 10.6% are gained on OVOBench for Qwen3-VL models at 4B, 8B, and 32B scales, respectively.The results support positive gains across the reported Qwen3-VL parameter scales.

E. Ablation Studies

The ablations show that curvature-based scoring and adaptive hierarchical memory allocation are central to CurveStream’s accuracy-efficiency gains and robustness. Dynamic allocation preserves semantic and spatial information while controlling token and computational costs.

  • Effectiveness of Curvature Metric: Curvature scoring significantly outperforms uniform and cosine-similarity motion sampling under identical visual token constraints.It also provides a lightweight geometric alternative to computationally expensive, pixel-noise-sensitive dense optical flow.
  • Adaptive Hierarchical Visual Memory Management: 100% clear memory depletes the context window and can trigger catastrophic forgetting of early memory.
  • Adaptive Hierarchical Visual Memory Management: 0% clear memory discards critical spatial details and causes a drastic performance drop.
  • Adaptive Hierarchical Visual Memory Management: Approximately 50% adaptive clear-memory allocation achieves the best accuracy while reducing computational overhead by about 40%.The content-aware hybrid mechanism dynamically balances native high-resolution keyframes with low-resolution transition frames.
  • Hyperparameter Robustness: The dual-threshold parameters k1 and k2 maintain stable accuracy and frame-sampling ratios across settings without tedious heuristic tuning.This supports CurveStream’s use as a robust plug-and-play module across diverse data streams.
  • Overall Findings: CurveStream establishes state-of-the-art streaming-video results while providing broad architectural compatibility and substantial baseline gains.

APPENDIX A CURVESTREAM ALGORITHM

CurveStream processes each incoming frame online, scores semantic transition intensity from feature motion and curvature, and routes frames into hierarchical memories under a fixed capacity. Qualitative cases illustrate how this routing preserves transient actions, causal context, and fine-grained spatial details.

  • Algorithm: The online scheduler extracts frozen-encoder features and evaluates each incoming frame with the Curvature-Aware Scorer without future context.The scorer combines first-order motion variation with second-order geometric curvature.
  • Qualitative Analysis: CurveStream assigns high-resolution memory to informative frames, preserving unobscured pot patterns and exposed forks for fine-grained recognition.These cases address failures caused by uniform downsampling and severe object occlusion.
  • Algorithm: The final Curvature Score is computed as motion variation plus λ-weighted geometric curvature.
  • Algorithm: The scheduler recursively updates the transient score distribution and generates dynamic dual thresholds for memory routing.
  • Algorithm: High-scoring frames or the query timestamp enter Clear Memory at high resolution, intermediate frames enter Blurred Memory at low resolution, and redundant frames are discarded.
  • Algorithm: Strict FIFO eviction removes the oldest tokens when the updated memory exceeds its maximum capacity.
  • Qualitative Analysis: In dynamic action recognition, preserving a sharp curvature peak for drinking avoids the baseline’s camera-setup hallucination.
  • Qualitative Analysis: For future prediction, retained interaction history preserves a coherent causal sequence and supports inferring smartphone operation despite background bias.

B. Differential Geometric Perspective of Ct

The geometric formulation defines curvature from changes in the direction of adjacent feature displacements rather than displacement magnitude alone. This makes the metric sensitive to directional semantic transitions while reducing sensitivity to smooth physical motion.

  • Discrete Curvature: Adjacent normalized displacement vectors T_1 and T_2 represent the feature trajectory’s local unit tangents.
  • Discrete Curvature: The geometric curvature C_t is defined as the cosine distance between adjacent displacement vectors.Equivalently, C_t = 1 - ⟨T_1,T_2⟩.
  • Geometric Equivalence: The squared distance between unit tangents is mathematically equivalent to twice the cosine-distance curvature.
  • Geometric Interpretation: C_t evaluates the directional derivative of feature evolution rather than scalar displacement alone.The derivation establishes equivalence up to a constant scaling factor with squared unit-tangent variation.

C. Theoretical Advantages of Semantic Decoupling

CurveStream’s curvature formulation suppresses constant-velocity motion noise while remaining sensitive to directional semantic shifts. Its hierarchical memory then uses these distinctions to retain salient events and support long-context streaming tasks.

  • Theoretical Advantages: During smooth camera panning, approximately parallel tangent vectors produce C_t ≈ 0, suppressing constant-velocity motion noise.
  • Theoretical Advantages: Sudden semantic shifts create directional changes in the feature trajectory, making them distinguishable from continuous physical motion.
  • Evaluation Context: The evaluation compares CurveStream with standard MLLMs and streaming baselines across StreamingBench and OVO-Bench.
  • Streaming-Benchmark Implications: Hierarchical memory routing provides persistent storage for salient events when redundant later frames would otherwise squeeze them from limited context windows.
  • Streaming-Benchmark Implications: Retaining action mutations as discrete keyframe snapshots compresses long videos into high-density sequences of core events for counting and summarization.

B. Analysis of Improvements on OVO-Bench

CurveStream improves streaming and offline video understanding by routing semantically important frames through curvature-aware hierarchical memory. Its mechanisms target fine-grained visual detail, temporal state transitions, changing spatial viewpoints, and transient motion across multiple benchmarks.

  • Streaming visual perception: High-resolution OCR and attribute recognition benefit from prioritizing frames with significant text or attribute changes under memory pressure.The Curvature-Aware Scorer supports hierarchical allocation instead of relying on global downsampling that can lose fine-grained information.
  • Streaming visual perception: Action recognition and future prediction retain action boundaries as key semantic nodes, preserving state-transition history for current-action reasoning and future evolution prediction.This reduces the risk of hallucination associated with context truncation.
  • Streaming visual perception: Adaptive K-Sigma routing better preserves frames containing rich spatial topological relations under variable camera-motion rhythms.This addresses the failure of fixed uniform sampling to retain optimal viewpoints for spatial and object understanding.
  • Offline video understanding: CurveStream generalizes to offline long-video understanding by estimating semantic information density globally and routing limited tokens toward highly dynamic segments.The framework was evaluated across FAVOR-Bench and MVBench.
  • Offline video understanding: MVBench’s temporal-dependency tasks benefit from retaining high-curvature state-mutation points in Clear Memory to construct a visual causal evidence chain.MVBench reports performance across 19 fine-grained subtasks.
  • Offline video understanding: CurveStream captures transient micro-motion while preserving a global view by placing local high-frequency changes in Clear Memory and broader context in Blurred Memory.This design is evaluated on FAVOR-Bench’s subtle camera-motion and environmental-change tasks.
  • Inference configuration: The inference policy uses a 20-frame maximum visual memory capacity and downsamples Blurred Memory frames to spatial resolution 224.These settings simulate stringent memory constraints while compressing transitional observations.

APPENDIX G ABLATION STUDY

The ablation study separates the contributions of the Curvature-Aware Scorer and Hierarchical Visual Memory Management, showing that their combination produces the strongest gains through complementary functions. CAS identifies semantic transitions, while HVMM preserves and compresses those selected states under a fixed token budget.

  • Ablation design: The ablations compare CAS and HVMM individually and jointly against passive uniform sampling with a FIFO cache on StreamingBench and OVO-Bench.The study evaluates both independent module effects and their combined behavior.
  • Ablation design: The ablation configuration and reported OVO-Bench comparisons are documented in Tables X and XI.Table X covers the OVO-Bench ablation, while Table XI lists core inference hyperparameters.
  • Individual modules: 9.12% and 8.39% average improvements result from adding CAS alone on StreamingBench and OVO-Bench, respectively.The result supports CAS sensitivity to feature-manifold curvature associated with semantic transitions.
  • Individual modules: 9.76% and 4.69% improvements result from adding HVMM alone on StreamingBench and OVO-Bench, respectively.HVMM uses Clear Memory and Blurred Memory to retain historical features without increasing the overall token budget.
  • Combined architecture: 12.04% and 10.66% total gains result when CAS and HVMM operate jointly on StreamingBench and OVO-Bench, respectively.The combined gain exceeds the sum of individual module improvements, including a 3.93% > −0.57%+1.68% example for OVO-Bench STU.
  • Combined architecture: CAS marks high-curvature transition points, while HVMM stores them in Clear Memory and compresses less critical observations into Blurred Memory.Without HVMM, selected frames may be evicted; without CAS, HVMM can degrade into rigid structural segmentation.
Loading 2603.19571v1…