PaperlayerView exact revision on arXiv

Source-linked AI summary

Proact-VL: A Proactive VideoLLM for Real-Time AI Companions

Weicai Yan, Yuhong Dai, Qi Ran, Haodong Li, Wang Lin, Tao Jin, Xing Xie, Hao Liao, Jianxun Lian

arXiv:2603.03447v4cs.CV

TL;DR

Real-time AI companions need low-latency video interaction together with autonomous and controlled speaking behavior. Proact-VL addresses this through a streaming framework and Live Gaming Dataset, and experiments report stronger response quality and timing while preserving video understanding. The paper presents this as practical for always-on interactive companions, within the scope of gaming-based evaluation and identified future-work boundaries.

  • Problem

    Existing streaming and proactive video models separately emphasize low latency or response timing, leaving controlled, short, continuous feedback under real-time constraints insufficiently addressed.

  • Method

    Proact-VL combines chunk-wise streaming, a lightweight proactive response mechanism, and stability-oriented training objectives, evaluated with a Live Gaming Dataset spanning three gaming settings.

  • Results

    Proact-VL significantly outperforms existing methods in response quality and timing while maintaining robust general video understanding.

  • Takeaways & Limitations

    The results demonstrate the practicality of proactive, real-time AI companions for interactive gaming applications.

  • Takeaways & Limitations

    Future work should improve evidence-grounded generation, high-fidelity real-time video perception, and update-aware entity understanding.

Abstract

from arXiv · show

Proactive and real-time interactive experiences are essential for human-like AI companions, yet face three key challenges: (1) achieving low-latency inference under continuous streaming inputs, (2) autonomously deciding when to respond, and (3) controlling both quality and quantity of generated content to meet real-time constraints. In this work, we instantiate AI companions through two gaming scenarios, commentator and guide, selected for their suitability for automatic evaluation. We introduce the Live Gaming Benchmark, a large-scale dataset with three representative scenarios: solo commentary, co-commentary, and user guidance, and present Proact-VL, a general framework that shapes multimodal language models into proactive, real-time interactive agents capable of human-like environment perception and interaction. Extensive experiments show Proact-VL achieves superior response latency and quality while maintaining strong video understanding capabilities, demonstrating its practicality for real-time interactive applications.

1. Introduction

Proact-VL addresses the need for low-latency, controlled, continuous feedback by combining streaming processing with proactive response control. It is evaluated across three gaming interaction settings and improves response timing and quality while preserving video understanding.

  • Motivation: Real-time AI companions must control when, how long, and how much they speak during continuous video interaction.Constant talking can disrupt users, whereas excessive silence can weaken companionship.
  • Motivation: Prior proactive models decide when to respond but often produce long answers, while real-time models reduce latency without explicit speaking control.These limitations create coarse temporal granularity or excessive talking under streaming inputs.
  • Dataset and scenarios: The Live Gaming Dataset contains 561 hours across 12 titles and three settings: solo commentary, co-commentary, and user guidance.It is released as a resource for training and evaluating proactive real-time companions.
  • Approach: Proact-VL unifies chunk-wise processing, a proactive response mechanism, and a stability-oriented multi-tier loss to control when and what to speak.The framework supports continuous video processing and autonomous response decisions based on visual and contextual cues.
  • Results: Proact-VL delivers strong gains in response quality and timing while preserving general video understanding, demonstrating practicality for always-on AI companions.The paper reports that it outperforms existing methods in both proactivity timing and quality.

2. Related Work

VideoLLM research has extended visual understanding to streaming interaction, but existing approaches separately emphasize low latency or proactive timing. The paper positions Proact-VL as addressing the resulting gap in speaking control.

  • VideoLLMs: VideoLLMs extend multimodal language models by projecting visual embeddings into language-model token space and integrating temporal information across frames.This paradigm supports video-grounded instruction following and reasoning.
  • Streaming video understanding: Streaming video methods interleave video and text chunks or optimize attention and caching to support online generation and low-latency responses.VideoLLM-online, LiveCC, and StreamingVLM represent this line of work.
  • Proactive interaction: Streaming methods often provide limited control over when to speak, while proactive methods typically learn response policies but generate relatively long outputs.The related-work distinction motivates jointly modeling timing and streaming generation.

3. The Live Gaming Dataset and Benchmark

The Live Gaming Dataset provides broad, curated English gameplay coverage and task-specific processing for commentary and guidance. Its benchmark construction combines clip-level evaluation with long-horizon streaming tests.

  • Video data collection: Figure 2 organizes the dataset by three data categories, 12 game titles, and their corresponding genres.The inner, middle, and outer rings encode these three levels respectively.
  • Video data collection: The dataset covers 561 hours of English commentary footage across 12 popular game titles and multiple genres.Videos were selected from professional tournament broadcasts and expert influencer channels and archived at 420p.
  • Data processing: A two-branch processing pipeline addresses speaker-text alignment challenges separately for commentator and guide roles.Commentary processing must handle overlapping background music and NPC dialogue.
  • Commentator data processing: Commentary data processing combines WhisperX-large-v3 transcription and speaker identification, filtering of non-human sounds, expressive labeling, and domain-specific polishing.The pipeline preserves pauses, laughter, phonetic elongations, and game-specific terminology while filtering inappropriate content.
  • Guide data processing: Guide data processing segments gameplay into five-minute clips, identifies player queries and temporal intervals, and generates frame-aligned descriptions for GPT-4.1 to refine.The refinement preserves temporal accuracy and semantic fidelity while improving instructional clarity.
  • Benchmark construction: The benchmark includes 3,014 clip-level samples and 10 full-length streaming videos spanning 30 minutes to 2 hours.The clip-level suite covers in-domain, common-scenario, and out-of-domain evaluation, while streaming tests measure long-horizon stability.

4. Methodology

Proact-VL processes streaming video in one-second chunks, maintains context incrementally, and separately controls response timing through a lightweight gate. Its training objectives target utterance quality, switching behavior, temporal stability, and speaking rate.

  • Chunk-wise input schema: The model receives one-second streaming chunks containing video, optional user queries, and environmental context, then produces aligned utterance segments online.Utterances can continue across consecutive chunks for continuous interaction.
  • Chunk-wise input schema: A persistent transformer KV cache preserves prior conditioning and generated tokens, while each utterance becomes context for the next time step.This supports incremental processing and sustained conversational coherence.
  • Proactive response mechanism: Proact-VL extracts the <|FLAG|> hidden state, applies a gated MLP and sigmoid to estimate speaking probability, and compares it with threshold τ.A triggered decision generates short text; otherwise the model outputs silence.
  • Training strategy: The response mechanism uses binary cross-entropy supervision for speaking decisions and transition-aware weighting to emphasize rare response-to-silence or silence-to-response switches.The training strategy treats response state as a sequence rather than independent per-second points.
  • Training strategy: Stability regularization suppresses jitter and constrains the average speaking rate to match the human commentary baseline.Its terms encourage smooth probabilities within persistent states and calibrate overall response frequency.
  • Streaming implementation: A dual-cache sliding-window mechanism supports unbounded streaming within a fixed context length by evicting older streaming tokens while retaining recent interactions.A reverse-RoPE correction is applied to avoid positional discontinuity after eviction.

5. Experiment

Experiments evaluate Proact-VL across live gaming commentary, common and unseen-game commentary, long-form streaming, and ablations. The framework leads in text and response quality, remains stable over extended streaming, and preserves general video understanding.

  • Text Quality: Proact-VL achieves the strongest text quality on Live Gaming Commentary, leading across overall, Solo Commentary, and Co-Commentary evaluations.It improves over GPT-4o, matches or exceeds Gemini 2.5 Pro on overall LLM-judged quality, and maintains higher CC.
  • Response Quality: Proact-VL achieves the best overall F1 for response quality, with particularly strong gains in Co-Commentary and Guidance.It outperforms commercial models on triggering accuracy while keeping TimeDiff low.
  • Common and General Commentary: Proact-VL achieves the strongest text quality on both Ego4D and unseen-game Black Myth: Wukong commentary.The results indicate fluent general guidance commentary and strong out-of-domain performance.
  • Common and General Commentary: Proact-VL improves proactivity timing and triggering in both domains, achieving the best overall alignment on Ego4D and strong performance on Black Myth: Wukong.On the unseen game, it matches or surpasses strong real-time baselines.
  • Live Gaming Streaming: Proact-VL maintains consistent text quality as the streaming horizon increases, while response quality mildly degrades before stabilizing.This demonstrates robust long-form behavior under sustained streaming inference.
  • Ablation: Removing either training-loss term degrades precision, recall, F1, and timing accuracy, with the largest ablation effect occurring without Lreg.Without Lreg, F1 decreases by 49.05 and TimeDiff increases by 15.09.
  • Response Mechanism: A threshold of 0.5 produces the most practical response pattern, balancing silence and commentary better than thresholds 0.1 and 1.0.Threshold 0.1 causes near-continuous triggering, whereas 1.0 results in all-silence behavior.
  • Efficiency: Per-token generation time remains essentially constant across streaming-window settings, and the system is expected to handle 10–15 FPS video streams with a fixed 0.3-second generation budget.Each frame is represented with 364 tokens.

6. Conclusion

The paper frames gaming commentary and guidance as concrete settings for human-like AI companions and introduces Proact-VL with the Live Gaming Dataset. Extensive experiments show improved response quality and timing over existing methods.

  • Conclusion: Proact-VL combines chunk-wise processing, a lightweight response mechanism, and specialized training objectives for real-time proactive interaction.The framework is supported by the newly proposed Live Gaming Dataset.
  • Conclusion: Extensive experiments demonstrate that Proact-VL significantly outperforms existing methods in both response quality and timing.

Impact Statement

Proact-VL targets safer, more accessible real-time AI companions while addressing deployment risks and streaming constraints through structured inputs, response gating, and cache management.

  • Impact and risks: Real-time AI companions could improve accessibility and engagement for live content, but misinformation and biased commentary require safety consideration.The dataset was cleaned to promote healthy narratives as a foundation for responsible deployment.
  • Response control: A response head uses the semantic-free <|FLAG|> marker to compute a stable response score instead of relying on decoding-sensitive silence-token probabilities.This avoids unintended semantic priors from reusing an existing token.
  • Response control: The decide-then-generate pipeline first scores whether to respond, then generates a short commentary clip only when the score exceeds a threshold.Otherwise, it emits a fixed silence placeholder.
  • Input structure: History, the current video chunk, and the user query are concatenated in that order to support continuity, coordination, and reference resolution.The query follows the video because it is semantically closer to the response.
  • Long-stream inference: When context capacity is approached, the system evicts the oldest 20% of streaming-cache tokens and applies reverse-RoPE correction to preserve recent context and positional coherence.The method is designed for continuous operation over arbitrarily long streams.
  • Long-stream inference: Monotonically increasing position ids can exceed frequently seen training ranges during long-running inference, weakening instruction following and destabilizing generation.Reverse RoPE re-bases the remaining cached positions to address this drift.

B.2. Benchmark Construction

The benchmark combines game-diverse training data with complementary clip-level and full-video test suites to evaluate commentary, guidance, and long-horizon streaming stability.

  • Data splits: The training set spans ten games, using an 80%/10%/10% video-wise split for training, testing, and future use.The ten games include Cyberpunk 2077, StarCraft II, Baldur’s Gate 3, Elden Ring, Tears of the Kingdom, Yu-Gi-Oh, League of Legends, CSGO, Street Fighter 6, and Minecraft.
  • Clip-level benchmark: Live Gaming Benchmark evaluates clip-level commentary and guidance, including an in-domain generalization game, Black Myth: Wukong.Its sampling stratifies clips by response rate across low, moderate, and high-response bins.
  • Streaming benchmark: Live Gaming Benchmark-Streaming uses one complete video per game, including 30-minute, one-hour, and two-hour videos, to assess sustained quality and stability.The set contains one 30-minute video, eight 1-hour videos, and one 2-hour video.
  • Data scale: The training set contains 128,000 samples from 12 sources, while evaluation includes game-centric clips, Minecraft length variants, and an Ego4D subset.The evaluation design combines game-domain coverage with general streaming data.

C.1. Evaluation Metrics

The evaluation combines text-quality, timing, and proactive-behavior metrics, while refining temporal scoring to capture both response coverage and false positives across the full video timeline.

  • Timing metrics: TimeDiff measures the temporal accuracy of predicted responses relative to ground-truth intervals.Predicted responses are matched by their start times within each annotated interval.
  • Timing metrics: The refined TimeDiff metric penalizes redundant or misaligned predictions outside an expanded tolerance interval.The expanded interval is Δ_i = [a_i − δ, b_i + δ], with α as the penalty coefficient.
  • Proactive metrics: PAUC integrates response quality along the temporal axis to capture cumulative impact and the dynamics of proactive behavior.The metric uses GPT-5.1 as judge and reports results with an initial score of 0 and ω = 0.5.
  • Proactive metrics: F1 evaluates the entire video as a binary temporal axis, jointly accounting for response recall and false positives.It derives precision and recall by comparing predicted response timelines with positive ground-truth intervals and negative remaining timestamps.
  • Text-quality metrics: LiveU measures second-level streaming usability, while FinalQ measures consolidated script quality after concatenating clip outputs.LiveU averages Time, Rate, and TextU; FinalQ averages Fidelity, Continuity, and Substance.
  • Game-wise analysis: Game-wise analysis reports response-quality and text-quality dimensions across games, with higher LiveU and FinalQ on almost all games.The authors interpret this pattern as generalization beyond a single title.

E. Robustness of LLM-as-a-Judge Evaluation

Proact-VL’s reported evaluation advantage remains consistent across alternative judge models and repeated judging, supported by pairwise human evaluation across three gaming scenarios.

  • Judge-model robustness: Proact-VL ranks first in LiveU and FinalQ across Solo, Co-Commentary, and Guidance under both evaluated judge models.It also obtains the highest win rate, despite minor metric-level reorderings among baselines.
  • Judge-model robustness: Replacing the judge model changes absolute scores but preserves the high-level conclusion that Proact-VL outperforms prior LiveCC baselines.This supports robustness to judge calibration differences.
  • Judge stochasticity: Across five repeated judging runs, win-rate variability is small in all three settings.The reported means, standard deviations, and 95% confidence intervals indicate stable results under judge stochasticity.
  • Human evaluation: The user study compares Proact-VL with one baseline from each Offline, Proactive, and Real-Time category across three scenarios.The scenarios are single-person commentary, multi-person commentary, and game instruction.
  • Human evaluation: Each pairwise comparison was independently assessed by three evaluators to establish inter-annotator reliability.Model outputs were rendered as synchronized subtitles with synthesized speech embedded in video clips.

G. Ablation Study for Training Data

The full training mixture performs best overall, with each source contributing strongly to its corresponding domain. Prompt ablations further show that response timing and commentary consistency are sensitive to model type and prompt strength.

  • Training-data ablation: The full training mixture achieves the best overall performance, while removing Gaming, Ego4D, or Live-SFT data lowers its corresponding domain score.Gaming CC drops 13.08% without Gaming data, Ego4D drops 22.39% without Ego4D data, and Livesports CC drops 7.69% without Live-SFT.
  • Training-data ablation: Fusing all three training sources preserves strong generalization across domains.The full model maintains strong generalization, with acceptable Livesports decreases and the strongest Ego4D results.
  • Prompt ablation: Full system-and-user prompt constraints can sharply reduce Solo F1 for LiveCC-Base, from 47.05 to 16.40.The result indicates that overly strong instructions can over-constrain base models and suppress timely responses.
  • Prompt ablation: LiveCC-Instruct shows only modest CC and F1 changes across prompt settings, suggesting greater robustness to prompt formatting.The model still benefits from lightweight contextual grounding in the user template.
  • Prompt ablation: The main experiments therefore use minimally invasive prompting with essential user-query and history context.The system prompt remains close to the backbone prompt, avoiding heavy task and persona constraints.

I. General Video Understanding Evaluation

Proact-VL largely preserves general video understanding after proactive fine-tuning, with clear Video-MME gains, comparable MVBench performance, and a modest LongVideoBench decrease. Additional analyses show threshold and context-window choices trade response coverage against consistency, while case studies demonstrate commentary and guidance behaviors alongside recognized failures.

  • General video benchmarks: MVBench accuracy remains comparable to Qwen3-VL at 64.7% versus 66.30%, with gains on Action Sequence and Object Interaction.Regressions occur on Moving Attribute, Moving Count, and Object Existence.
  • General video benchmarks: Video-MME overall scores improve for Qwen3-VL from 57.3 to 59.1 without subtitles and from 58.3 to 59.8 with subtitles.For Qwen2.5-VL, scores improve from 53.1 to 57.1 without subtitles and from 54.8 to 58.4 with subtitles.
  • General video benchmarks: LongVideoBench changes from 54.5 to 52.8 overall, despite improvement from 64.0 to 68.6 on the 60-second split.The remaining duration splits show slight decreases.
  • General video benchmarks: Across MVBench, Video-MME, and LongVideoBench, Proact-VL shows no systematic degradation in general video understanding.It gains on Video-MME, remains comparable on MVBench, and decreases modestly on LongVideoBench.
  • Hyperparameter analysis: Increasing the response threshold consistently lowers F1 while favoring higher-consistency CC in the three interaction settings.CC peaks around 0.9 for Co-Commentary, 0.6 for SOLO, and 0.5 for Guidance.
  • Hyperparameter analysis: Larger context windows generally improve CC while leaving F1 relatively stable, with best Guidance CC/F1 at 16384: 43.02/51.07.SOLO CC peaks around 24576 at 55.08, and Co-Commentary CC reaches 56.98 at 24576.
  • Scenario cases: Case studies show Proact-VL performing trade-off-aware RPG commentary, anticipating viewer confusion, coordinating with a co-commentator, and timing Minecraft guidance to hazards.The guidance example includes an inventory-readiness prompt before lava-conversion instructions.
  • Failure cases: Failure cases expose brittle HUD arithmetic and visual grounding, including a hallucinated 2K lead and repetitive fillers in cluttered interfaces.The authors connect the first failure to OCR and numerical reasoning limitations.

L. Human Alignment Analysis

The human-alignment study evaluates whether LLM-based scores track human preferences and distributions across 100 instances and three methods. LiveU shows strong pairwise agreement, while calibration exhibits method-specific bias, including overestimation of Proact-VL.

  • Study design: The validation study covers 100 sampled instances and 300 outputs from Livecc-7B-Base, LiveStar, and Proact-VL.It compares LLM-based evaluations with human judgments using pairwise preference and score-calibration analyses.
  • Score distributions: Figure 17 compares human ratings, LiveU scores, and FinalQ scores across the sampled instances for each method.The distributions provide a visual view of score alignment and spread.
  • Pairwise alignment: LiveU pairwise preference accuracy reaches 93.8% for Livecc-7B-Base versus LiveStar and 84.7% for Livecc-7B-Base versus Proact-VL.The analysis uses strict accuracy and excludes tied human ratings.
  • Calibration: LLM scores are well calibrated for Livecc-7B-Base with mean bias −0.030, but underestimate LiveStar and overestimate Proact-VL.The study reports mean bias, MAE, RMSE, KS distance, and 1D Wasserstein distance.

M. Limitations and Future Work

The paper identifies evidence grounding, temporal video fidelity, and update-aware entity recognition as limitations for practical streaming companions. It proposes improving generation grounding, high-fidelity perception, and robust entity understanding in future work.

  • Evidence grounding: Commentary can remain fluent but only weakly correlated with on-screen evidence, producing plausible yet poorly grounded narration.The authors identify fine-grained visual grounding and reduction of hallucinated or generic narration as open directions.
  • Video fidelity: Processing sparse frames such as 2 FPS can miss transient cues and make fast actions, UI changes, and short-lived events harder to understand.The paper calls for more efficient streaming encoders and memory mechanisms for higher FPS and resolution.
  • Entity understanding: In-game character, role, and entity identification remains brittle because the model may rely on internal knowledge rather than reliable on-screen recognition.Frequent game updates can introduce new versions, characters, and items that challenge this reliance.
  • Future work: Future work should jointly improve evidence-grounded generation, high-fidelity streaming perception, and update-aware entity understanding.These directions are framed as requirements for better practical AI companions for streaming commentary.
Loading 2603.03447v4…