Source-linked AI summary
PEARL: Personalized Streaming Video Understanding Model
Yuanhong Zheng, Ruichuan An, Xiaopeng Lin, Yuxing Liu, Sihan Yang, Huanyu Zhang, Haodong Li, Qintong Zhang, Renrui Zhang, Guopeng Li, Yifan Zhang, Yuheng Li, Wentao Zhang
TL;DR
Existing personalized VLM methods largely address static images or offline videos, leaving streaming, timestamped, interactive personalization underdeveloped. The paper defines PSVU, introduces PEARL-Bench, and proposes PEARL, a training-free memory-and-retrieval framework; PEARL consistently improves performance across architectures and achieves state-of-the-art results.
Problem
Personalization methods remain largely limited to static images or offline videos, whereas real-world assistants require streaming visual inputs and interactive personalized responses.
Method
The paper defines PSVU, introduces PEARL-Bench for frame-level and video-level timestamped evaluation, and proposes PEARL with dual-grained memory and concept-aware retrieval.
Results
PEARL consistently achieves state-of-the-art performance across multiple architectures and improves the video-level setting, with Qwen3-VL-8B+PEARL outperforming ReKV by 24.28%.
Takeaways & Limitations
PEARL provides a training-free, plug-and-play strategy for personalized understanding of continuous video streams.
Abstract
from arXiv · showhide
Human cognition of new concepts is inherently a streaming process: we continuously recognize new objects or identities and update our memories over time. However, current multimodal personalization methods are largely limited to static images or offline videos. This disconnects continuous visual input from instant real-world feedback, limiting their ability to provide the real-time, interactive personalized responses essential for future AI assistants. To bridge this gap, we first propose and formally define the novel task of Personalized Streaming Video Understanding (PSVU). To facilitate research in this new direction, we introduce PEARL-Bench, the first comprehensive benchmark designed specifically to evaluate this challenging setting. It evaluates a model's ability to respond to personalized concepts at exact timestamps under two modes: (1) Frame-level, focusing on a specific person or object in discrete frames, and (2) a novel Video-level, focusing on personalized actions unfolding across continuous frames. PEARL-Bench comprises 132 unique videos and 2,173 fine-grained annotations with precise timestamps. Concept diversity and annotation quality are strictly ensured through a combined pipeline of automated generation and human verification. To tackle this challenging new setting, we further propose PEARL, a plug-and-play, training-free strategy that serves as a strong baseline. Extensive evaluations across 8 offline and online models demonstrate that PEARL achieves state-of-the-art performance. Notably, it brings consistent PSVU improvements when applied to 3 distinct architectures, proving to be a highly effective and robust strategy. We hope this work advances vision-language model (VLM) personalization and inspires further research into streaming personalized AI assistants. Code is available at https://github.com/Yuanhong-Zheng/PEARL.
1 Introduction
The paper introduces Personalized Streaming Video Understanding (PSVU) to address personalization methods’ restriction to static images and offline videos. It also presents PEARL-Bench and PEARL, a training-free framework designed for timestamped, interactive streaming understanding.
- Existing personalization methods mainly target static image-text tasks or offline, single-turn videos, unlike continuous real-world streams.
- PSVU models continuous visual input, dynamically defined concepts, and real-time interactive responses for personalized assistants.
- PEARL-Bench evaluates timestamped personalization through frame-level entities and video-level actions unfolding across continuous frames.
- PEARL is a training-free, plug-and-play framework that separates concept-centric knowledge from stream-centric observations and retrieves relevant historical evidence.
- PEARL achieves state-of-the-art results against 8 offline and online video understanding methods and remains robust across multiple architectures.
2 Related Works
Prior VLM personalization research spans personalized image understanding, generation, and video understanding, with image personalization receiving most attention.
- Existing VLM personalization efforts broadly cover personalized image understanding, personalized generation, and personalized video understanding.
- Personalized image understanding has predominantly used finetuning-based approaches, alongside other personalization paradigms.
3 PEARL-Bench
PEARL-Bench defines streaming personalization around evolving user concepts, timestamped queries, and frame-level or video-level understanding. Its benchmark combines diverse videos, multiple QA types, and automated filtering with human verification.
- 3.1 Task Definition: PSVU treats streaming video as continuous scenes where users introduce concepts at arbitrary timestamps for later querying.
- 3.1 Task Definition: Frame-level concepts are static entities from single frames, whereas video-level concepts are dynamic actions unfolding across continuous clips.
- 3.1 Task Definition: The benchmark includes Concept-Definition, Real-Time, and Past-Time QA, requiring registration, present-scene grounding, or historical evidence retrieval.
- 3.2 Benchmark Overview: PEARL-Bench supports long-form streaming video, multi-turn interaction, and both personalization modes, comprising 132 videos and 2,173 timestamped annotations.
- 3.3 Curation Pipeline: The curation pipeline collects and filters videos, annotates three QA types, and applies quality control using automated filtering plus human verification.
- 3.3 Curation Pipeline: Concept-definition annotations register named entities or actions using visual evidence and textual descriptions, while names are randomized to reduce reliance on prior knowledge.
4 PEARL Framework
PEARL addresses PSVU by dynamically defining concepts in streaming video and retrieving personalized concepts and visual context for real-time responses. Its training-free framework combines dual-grained memory with concept-aware retrieval.
- PEARL dynamically defines user concepts at timestamps and answers subsequent queries in real time.
- The task requires constructing answers from query-relevant concepts and visual context while managing unbounded history and evolving concepts.
- Dual-grained memory separates concept-centric knowledge from stream-centric observations, retaining concepts and long-range video evidence.
- Concept Memory entries contain a name, visual evidence, and textual description generated from the current clip.
- Concept-aware retrieval rewrites concept names into descriptions, selects top-K historical clips by embedding similarity, expands them temporally, and combines them with the current clip.
5 Experiments
Experiments show that PEARL consistently improves offline and online video models, including on the more difficult video-level setting. Its memory and query-rewriting components drive accuracy gains while adding relatively small retrieval overhead compared with model inference.
- Comparison with Offline Baselines: PEARL improves three offline base models by 8.55%, 9.36%, and 23.47%, demonstrating gains across distinct architectures.
- Comparison with Online Baselines: PEARL variants surpass the strongest online baseline StreamForest-7B by 3.01%, 3.84%, and 17.22%.
- Video-level Results: Video-level scores are lower than frame-level scores, while Qwen3-VL-8B+PEARL exceeds ReKV by 24.28% on video-level accuracy.
- Effectiveness of PEARL Design: Adding Concept Memory increases Real-Time accuracy by over 35%, while Streaming Memory increases Past-Time accuracy by more than 20%.
- Effectiveness of PEARL Design: Query Rewriting further raises average accuracy by 4.28% over the non-rewritten version.
- Efficiency of PEARL Design: LLaVA-OV-7B+PEARL remains faster than all online baselines while improving frame-level average accuracy by 8.55%.
- Efficiency of PEARL Design: PEARL retrieval and rewriting add low, model-invariant latency, whereas underlying LLM inference is the primary bottleneck.
- Hyperparameter Analysis: Past-Time accuracy improves rapidly as K increases and plateaus after K ≥3, indicating moderate retrieval is sufficient.
6 Conclusion
PEARL introduces a benchmark and framework for personalized streaming video understanding, covering frame-level and video-level concepts in interactive timestamped queries. The benchmark organizes concept definitions and real-time and past-time questions into fine-grained categories, while PEARL supports retrieval and reasoning over personalized visual content.
- Conclusion: PEARL-Bench evaluates personalized streaming video understanding through frame-level and video-level personalization in streaming videos.Its questions are organized into Concept-Definition QA, Real-Time QA, and Past-Time QA categories.
- QA Sub-categories: Concept-Definition QA introduces new user-defined concepts at specific timestamps using direct or contextual descriptions.Frame-level definitions identify people or objects, while video-level definitions identify personalized action sequences unfolding across continuous clips.
- QA Sub-categories: Real-Time QA queries established concepts in the current scene across presence, behavior, appearance, location, relation, and action sub-tasks.These questions assess immediate perception without requiring historical retrieval.
- QA Sub-categories: Past-Time QA tests historical evidence retrieval through event-based and time-based questions about established concepts.Event-based queries rely on a specific event, whereas time-based queries require understanding the temporal order of events.
- Question Templates: Question templates define variables such as concept names, features, objects, locations, events, and standardized definition suffixes for annotation.The suffixes enforce consistent naming of frame-level identities and video-level motion patterns.
- Implementation: PEARL generates concise descriptions for retrieval by emphasizing stable frame-level features and using the original concept description to identify the target.The prompts focus on persistent attributes rather than changing clothing, accessories, poses, or surroundings.
- Visualization: The visualization demonstrates accurate real-time recognition and long-term temporal retrieval for personalized concepts at specific timestamps.Examples include recognizing Nuriya’s presence and posture, locating Kavery, and identifying who saved Truz.
D.2 Video-level Visualization
The video-level visualization shows PEARL registering personalized action sequences during an ongoing stream and recognizing them later across changing performers and contexts.
- Video-level Visualization: PEARL registers multiple user-defined action sequences as video-level concepts during the initial stream.The example defines Action A, Action B, and Action C at [00:04], [00:16], and [00:34].
- Video-level Visualization: PEARL later recognizes customized actions when different characters perform them in different contexts.The example identifies Action A at [01:08], Action B at [01:30], and Action C at [03:25] for the person on the right wearing blue clothes.
E Detailed Implementation Settings
The implementation uses a multimodal embedding model and scene detection to represent streaming clips and segment them into temporally coherent units.
- Multimodal Embedding Model: PEARL uses Qwen3-VL-Embedding-2B to encode generated visual descriptions and streaming video clips into a unified feature space.This embedding model supports retrieval across the concept and stream representations.
- Scene Detection: PySceneDetect segments the stream by detecting fast cuts from HSV color changes between adjacent frames.The threshold is 27.0, with clip durations constrained between 1.0 and 8.0 seconds; longer scenes are split proportionally.
E.2 Cyclic Option Rotation Evaluation Strategy
The evaluation rotates the correct answer across all four option positions to reduce sensitivity to multiple-choice answer placement.
- Cyclic Option Rotation: Each multiple-choice question is evaluated four times with the correct answer rotated through positions A, B, C, and D.The contents of the originally correct option are swapped with the target position while other distractors remain unchanged.
F.1 Hyperparameter Analysis for Real-Time QA
Real-Time QA is relatively insensitive to retrieval settings, but modest historical context can help before excessive history introduces distracting noise.
- Accuracy fluctuations remain within 5% across different K and N settings.
- Retrieving a small amount of historical clips generally improves performance slightly over K = 0.
- Increasing K and N further produces a downward accuracy trend as irrelevant historical information interferes with real-time judgment.
F.2 Effect of Model Scales
PEARL improves performance consistently across evaluated model scales and architectures, while scaling offline models alone does not substantially address PSVU’s visual-context requirements.
- Experimental setup: The evaluation covers Qwen2-VL models at 2B and 7B and Qwen3-VL models at 4B and 8B, with and without PEARL.
- Robustness across model scales: 18.00% and 23.47% average-accuracy gains occur for Qwen3-VL 4B and 8B, respectively, with PEARL.
- Robustness across model scales: 4.17% and 9.36% average-accuracy gains occur for Qwen2-VL 2B and 7B, respectively, with PEARL.
- Paradigm mismatch for offline models: Increasing offline-model scale does not produce significant performance gains, indicating a paradigm mismatch for PSVU.