PaperlayerView exact revision on arXiv

Source-linked AI summary

SyncPro: Synchronous Reasoning and Responding for Proactive Streaming Video Understanding

Xueyun Tian, Bingbing Xu, Jiaming Liu, Jinpeng Yu, Yang Shi, Yu Zhao, Ruihua Huang, Yuanzhuo Wang, Huawei Shen

arXiv:2610.32132v1cs.CVcs.CLcs.LG

TL;DR

Proactive streaming video understanding requires coordinating evidence tracking with timely responses, a problem not fully addressed by existing methods. SyncPro uses independently activated Inner and Response streams with shared causal history, and experiments report improved response timing and content across three benchmarks while largely preserving offline video understanding.

  • Problem

    Proactive assistants must track unfolding, query-relevant evidence and reassess response conditions, while coordinating internal monitoring with user-facing responses.

    Source: §1 · pp. 1–2

  • Method

    SyncPro uses specialized Inner and Response branches that activate independently from shared history, trained on 189K dual-stream examples with supervised fine-tuning and reinforcement learning.

    Source: §1 · p. 2

  • Results

    Across three benchmarks, SyncPro improves response timing and content while largely preserving offline video understanding; on StreamingBench Proactive Output it reaches 41.2% accuracy within two seconds, 3.2 points above the strongest compared baseline.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The results support explicit reasoning and role separation for proactive video assistants.

    Source: §5 · p. 10

    Deployment in privacy-sensitive settings requires appropriate consent, access control, and bias assessment.

    Source: Ethics statement · p. 11

Abstract

from arXiv · show

Proactive streaming video understanding requires assistants to track unfolding events and respond when a user's requested condition is met. To make these decisions, assistants must accumulate relevant evidence and reassess response conditions as new observations arrive. Existing work offers complementary capabilities for this process: proactive video assistants learn when to respond, while streaming video reasoning methods generate scheduled intermediate text to organize observations and support subsequent answers. Together, these capabilities provide a foundation for proactive reasoning, but how to coordinate ongoing reasoning with timely responses remains underexplored. In a direct adaptation, fixed reasoning intervals can trigger updates when nothing relevant has changed or postpone useful updates until the next scheduled step. Sharing one generation path also couples two distinct requirements: preserving evidence for future decisions and producing only the response requested by the user. Moreover, responses generated after reasoning must wait for it to finish, even when sufficient evidence is already available. To address these challenges, we introduce SyncPro, a synchronous dual-stream framework with specialized generation paths for evidence tracking and user-facing responses. Its Inner and Response branches share causal history and activate independently, allowing relevant evidence changes to trigger internal updates and responses to proceed concurrently with ongoing reasoning. We construct 189K dual-stream examples and train SyncPro through supervised fine-tuning followed by trajectory-level reinforcement learning. Experiments across three proactive benchmarks show improved response timing and more accurate answers to user requests.

1. Introduction

Proactive streaming assistants must coordinate evidence monitoring with timely responses, but fixed schedules and a shared generation path do not accommodate their distinct roles and timing. SyncPro uses independently activated, specialized streams and reports gains across three proactive benchmarks.

1. Introduction

  • Proactive video assistants must track query-relevant evidence and reassess response conditions as events unfold, coordinating internal monitoring with user-facing answers.Prior work offers complementary capabilities in learning response timing and generating intermediate streaming reasoning, but their coordination remains a challenge.Source: §1 · p. 1
  • Fixed reasoning schedules can trigger redundant updates or delay useful ones, while reasoning and responses need distinct roles and independent activation.Responses supported by available evidence can proceed alongside ongoing reasoning.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • SyncPro uses specialized Inner and Response branches that independently decide whether to generate from shared history and the current observation.The branches track evidence and produce user-facing responses, respectively.Source: §1 · p. 2
  • SyncPro reaches 41.2% accuracy within a two-second response window on StreamingBench Proactive Output, 3.2 percentage points above the strongest compared baseline.Across OVO-Bench and ProactiveVideoQA, it also reports improved benchmark results.Source: §1 · p. 2
  • The authors construct 189K dual-stream examples and train SyncPro with supervised fine-tuning and reinforcement learning.Their contributions also describe per-step supervision for both streams and trajectory-level reinforcement learning to refine response timing.Source: §1 · p. 2

2. Related Work

Prior work addresses proactive response timing, streaming reasoning, and concurrent multimodal interaction, but these approaches retain coupling between reasoning and responding. SyncPro separates event monitoring from condition-triggered task execution through independently activated streams.

2. Related Work

  • Proactive video methods regulate response timing, while streaming reasoning methods generate intermediate text as observations arrive; ThinkStream combines them but reasons at every input step in a serial sequence.These approaches provide complementary foundations for proactive streaming video understanding.Source: §2 · pp. 2–3
  • Concurrent multimodal work spans parallel semantic and acoustic output, perception during response generation, and asynchronous or role-specific internal coordination.These examples differ in the levels at which they introduce concurrency.Source: §2 · p. 3
  • SyncPro separates ongoing event monitoring from condition-triggered task execution, with streams sharing causal history but activating independently without access to each other’s current outputs.Unlike the cited concurrent interaction approaches, its two streams serve distinct monitoring and task-execution roles.Source: §2 · p. 3

3. Method

SyncPro coordinates evidence tracking and user-facing responses through independently activated branches that share a causal history. The method pairs dual-stream trajectory construction with supervised fine-tuning and trajectory-level reinforcement learning.

3.1. Task Formulation

  • Inner tracks query-relevant evidence while Response produces the requested output; both may activate independently and update shared history at micro-turn boundaries.Only active Response content is returned to the user.Source: §3.2 · p. 3, §3.1 · p. 3

3.2. SyncPro Model Architecture

  • A shared multimodal backbone uses separate LoRA routes, independently decodes both branches, streams only Response content, and merges their states before discarding frame tokens.Both outputs remain in the history for later decisions.Source: Figure 2 · p. 4, §3.2 · p. 4

3.3. Dual-Stream Data Construction

  • Two construction routes synthesize proactive trajectories and adapt temporal annotations; 189,403 trajectories are retained after testing whether Inner entries reconstruct later responses from restricted visual context.The corpus spans 12 sources, three functions, and eight subtypes.Source: §3.3 · pp. 4–5, Figure 3 · p. 5

3.4. Two-Stage Training

  • Training proceeds in two stages: supervised fine-tuning teaches when and how to reason and respond, followed by trajectory-level reinforcement learning to improve response timing.RL trains on complete causal rollouts, optimizing event coverage while penalizing unmatched responses.Source: §3.4 · p. 5, §3.4.2 · p. 6

3.4.1. Supervised Fine-Tuning

  • Supervised fine-tuning combines branch-specific action and active-content losses with Action Temporal Ranking, which favors response actions at annotated times over nearby silent positions.Only the two LoRA adapters and action-token parameters are updated.Source: §3.4.1 · p. 6

3.4.2. Trajectory-Level Reinforcement Learning

  • Trajectory-level RL matches predicted response times to ordered ground-truth windows, uses hard-window F1 as reward, and standardizes rewards within balanced trajectory groups for policy optimization.The terminal timing advantage weights every token generated by both branches in the paired rollout.Source: §3.4.2 · p. 6

4. Experiments

Experiments compare SyncPro with prior methods and alternative formulations, then examine its timing, branch specialization, synchronous decoding, and retention of general video capabilities. Across three proactive benchmarks, SyncPro-RL improves response performance.

4.1. Experimental Setup

  • The setup follows official frame-indexed protocols, using Acc@2s and joint timing-and-content metrics for StreamingBench, three OVO-FAR tasks, and PAUC for ProactiveVideoQA.VST and ThinkStream are evaluated at 2 fps with released implementations and protocols; the SyncPro variants use 1 fps.Source: §4.1 · p. 7

4.2. Main Results

  • SyncPro-RL reaches 41.2% Acc@2s on StreamingBench-PO, 34.19 mean F1 on OVO-Bench Forward Active Responding, and 32.43 PAUC on ProactiveVideoQA VAD.These results exceed the strongest reported baseline by 3.2 points on StreamingBench-PO and 3.20 points on OVO-Bench; SyncPro-RL leads the compared methods on VAD.Source: §4.2 · pp. 7–8, Table 2 · p. 8

4.3. Formulation Comparisons

  • Under matched data and decoding budgets, SyncPro-SFT raises Joint@2s from 10.4% to 30.8% and Content Acc.@Response from 50.85% to 79.37% versus Response-only, and exact response accuracy from 8.98% to 45.24% versus Interleaved.Response-only removes internal-update targets; Interleaved retains updates but couples branch decisions in one activation sequence.Source: Table 3 · p. 8, §4.3 · p. 8
  • RL raises Joint@2s from 30.8% to 33.6% while Content Acc.@Response remains 79.14%; adaptive Inner triggers can skip redundant scheduled updates.Scheduled reasoning showed repeated text patterns and redundant Inner updates after the requested response.Source: §4.3 · pp. 8–9

4.4. Dual-Stream Mechanisms and Capability Retention

  • From step 500 to 1000, Inner activations fall from 113 to 93 while exact matches to annotated updates rise from 53 to 56; random timing yields 14.3 matches on average.This supports timing selectivity beyond activation frequency.Source: §4.4 · p. 9
  • Adapter swaps raise reference-token negative loglikelihood by 0.706 for Inner and 1.578 for Response, while concurrent and Inner-first decoding agree on all 1,870 held-out examples.The adapter result supports distinct branch roles; matching outputs preserve generation behavior across the tested decoding orders.Source: §4.4 · p. 9
  • Both SyncPro variants remain within 2.0 points of Qwen3.5-4B across three offline benchmarks and retain more MVBench accuracy than Interleaved and Response-only.The authors characterize this as largely preserving general video understanding after proactive adaptation.Source: §4.4 · p. 9

5. Conclusion

The conclusion presents SyncPro as a dual-stream approach to coordinating ongoing reasoning with timely responses, trained on 189K examples with supervised fine-tuning and reinforcement learning. The authors report improvements across three proactive benchmarks while largely preserving offline video understanding.

5. Conclusion

  • SyncPro coordinates ongoing reasoning with timely responses through specialized Inner and Response streams that share causal history, activate independently, and decode concurrently.The authors report improved response timing and content across three benchmarks while largely preserving offline video understanding.Source: §5 · p. 10

AI use statement

  • The authors report using Gemini 3.1 Pro Preview, Qwen-VL-Max-Latest, and GPT-5.4 in the synthetic-data pipeline, and AI tools for language editing.They state that the authors designed the pipeline, conducted the experiments, verified the claims and citations, and retain responsibility for the manuscript.Source: AI use statement · p. 11

Ethics statement

  • The work collects no new human-subject data and uses established public datasets under their released terms; privacy-sensitive deployment requires consent, access control, and bias assessment.The authors identify no additional ethical concerns beyond those associated with the source datasets and proactive video systems.Source: Ethics statement · p. 11

Reproducibility statement

  • The authors state that they will release the complete prompts, data, training and evaluation code, configurations, and model checkpoints needed to reproduce the experiments.Section 3 specifies the model, objectives, and data pipeline; Section 4 defines the evaluation protocol.Source: Reproducibility statement · p. 11

A. Implementation Details

Implementation uses a shared Qwen3.5 backbone with separate branch adapters and state transitions, followed by supervised fine-tuning and timing-focused reinforcement learning. The appendix also reports a merge-weight comparison and specifies runtime and data-generation procedures.

Supervised fine-tuning

  • SFT freezes the vision encoder and shared backbone, trains rank-64 LoRA adapters for each branch, and optimizes action-token vocabulary parameters.Training uses AdamW, one epoch, and bfloat16; SFT raises validation content-token accuracy by 5.86 points for Inner and 4.48 for Response, with action-event F1 of 0.71 and 0.61.Source: A. Implementation Details · p. 17

Branch state and positions

  • A block-causal mask lets each branch attend to the shared prefix and its own earlier tokens while blocking attention between current branches; completed outputs enter the next micro-turn’s shared state.Inference uses the same token order, mask, position IDs, and state transition as training; linear-attention layers fork the recurrent state and merge it before the next turn.Source: A. Implementation Details · p. 17

Reinforcement learning

  • RL starts from the SFT checkpoint, updates only branch-specific adapters, and uses a timing-only reward; one pass covers 800 examples balanced across four behavior groups.Pilot runs adding a content term increased response frequency and verbosity without improving conditional content accuracy.Source: A. Implementation Details · p. 17

Merge-weight comparison

  • Fixed symmetric merging at γ=0.5 outperforms learned per-branch weights overall under the evaluated SFT configuration: Acc@2s falls from 38.8% to 38.0%, and OVO-FAR mean F1 from 32.37% to 30.71%.Learned weights improve REC F1 by 3.14 points but reduce CRR and SSR F1 by 5.45 and 2.66 points, respectively.Source: A. Implementation Details · p. 17, Table 7 · p. 18

Streaming inference

  • At each micro-turn, the runtime batches both branches’ greedy decoding from shared state, merges their states, and streams only active Response content to the user.It samples video at 1 fps, encodes each frame once into at most 128 visual tokens, and reuses the KV cache while evicting frame tokens after their state update.Source: A. Implementation Details · p. 18

Data construction

  • Training examples pair queries and videos with per-micro-turn branch actions and text targets, synthesized from official training splits under a prefix-only constraint.Complete videos are used only to audit event coverage and trigger timing; refined candidates are regenerated under the same prefix constraint.Source: A. Implementation Details · p. 18

B. Long-Stream Runtime Profile

The long-stream profile measures memory and per-step latency for a single 480-second video stream at 1 fps, with compilation-affected steps excluded. It finds stable persistent allocation but growing transient memory and decoding cost as retained text grows.

Long-stream runtime profile

  • Across 478 measured steps, mean end-to-end latency is 985.3 ms and 74.9% of frames meet the 1-s budget; retained full-attention KV grows from 2,126 to 11,220 tokens.Persistent allocation changes by 1.1 MiB, while per-frame peak memory rises by 0.88 GiB to 11.75 GiB. Median latency is 827.2 ms and p90 latency is 1,596.7 ms.Source: B. Long-Stream Runtime Profile · pp. 18–19, Figure 6 · p. 19
  • With branch output lengths fixed, dual-branch CUDA latency rises by 15.6 ms per 1k retained KV tokens for Response-only activation and 32.4 ms for joint activation.The profile attributes growing transient peak memory and decoding cost to the increasing retained prefix, while fixed-capacity cache allocation keeps persistent memory stable.Source: B. Long-Stream Runtime Profile · p. 19, Figure 6 · p. 19

C. Detailed Benchmark Results

SyncPro’s trajectory-level reinforcement learning improves results across StreamingBench and OVO-Bench, with gains varying by task. On OVO-Bench, the backward-tracing average shifts slightly downward while other reported metrics improve or remain stable.

StreamingBench

  • 70.44 to 70.84: trajectory-level optimization raises StreamingBench’s overall score, improving six of ten tasks, with the largest gains on EU and CR.The other four task scores remain unchanged.Source: C. Detailed Benchmark Results · p. 19

OVO-Bench

  • On OVO-Bench, both SyncPro variants attain the strongest online FPD score of 76.24, while SyncPro-SFT leads online models on ASI at 68.24.RL maintains Real-Time Visual Perception average accuracy at 61.17 versus 61.13 for SFT, improves STU by 1.12 points, and leaves OCR, ACR, FPD, and OJR unchanged.Source: C. Detailed Benchmark Results · p. 19, Table 9 · p. 20
  • On OVO-Bench, the backward-tracing average shifts slightly from 43.95 to 43.62 after reinforcement learning.Tables 8 and 9 provide task-level results and frame metadata for the broader benchmark suites.Source: C. Detailed Benchmark Results · p. 19

D. Analysis of Inner Update Timing

The timing analysis finds that later training checkpoints align Inner updates better with annotated updates while producing fewer activations. Temporal tolerance recovers mostly neighboring-turn matches, and model timing outperforms random timing at matched activation counts.

Evaluation setup

  • The analysis covers 50 evaluated rollouts with 79 annotated Inner updates; exact matching requires the same micro-turn, while tolerant matching permits a displacement of up to two.Precision, recall, and F1 pool event counts across cases; Figure 5 reports mean per-case F1.Source: D. Analysis of Inner Update Timing · p. 21, Table 10 · p. 21

Learning to select update times

  • From step 500 to 1000, activations fall from 113 to 93 while exact matches rise from 53 to 56; precision improves by 13.3 points and recall by 3.8 points.The later checkpoint achieves better alignment with fewer activations.Source: D. Analysis of Inner Update Timing · p. 21

Temporal tolerance

  • At step 1000, one-micro-turn tolerance increases matches from 56 to 68, while a second micro-turn adds one match.Of 69 matches within two micro-turns, 56 are exact, eight occur one step early, four one step late, and one two steps early.Source: D. Analysis of Inner Update Timing · p. 21

Random timing with matched activation counts

  • At matched activation counts, random timing averages 14.3 exact matches and 33.2 within one micro-turn, versus 56 and 68 for the model.This comparison supports temporal alignment beyond that obtained from random timing at the same activation count.Source: D. Analysis of Inner Update Timing · p. 21

E. Data-Construction Prompts

The data-construction prompts specify independent internal and external decisions, criteria for proactive requests, and rules for synthesizing and revising response trajectories.

E. Data-Construction Prompts

  • The execution prompts are formatting-normalized abridgements, with bracketed ellipses marking omitted explanations and few-shot examples.The released artifacts will include the complete execution prompts.Source: E. Data-Construction Prompts · p. 22

A1. Dual-stream model interface (abridged)

  • At each one-fps video chunk, the assistant makes independent internal and external decisions, and either, both, or neither branch may activate.The output contains one decision per branch; only speak content is visible to the user.Source: A1. Dual-stream model interface… · p. 22
  • Internal updates record only new query-relevant evidence or state changes, using concise summaries and silence when no useful change occurs.Stored facts should not be repeated.Source: A1. Dual-stream model interface… · p. 22

A2. Proactive-query construction and filtering (abridged)

  • A proactive request must depend on future visual evidence, need intermediate state updates, and have a precise observable trigger before stream termination.Requests are filtered if they require a terminal condition, audio, an unanswerable query, or external knowledge.Source: A2. Proactive-query construction and… · p. 22
  • Retained requests are returned with query, interaction, reasoning, tracking, and response fields, while filtered requests include a category and evidence-based reason.The output also includes plausible answer times for retained requests.Source: A2. Proactive-query construction and… · p. 22

A3. Causal dual-stream trajectory synthesis (abridged)

  • Trajectory synthesis tracks minimal sufficient query-relevant state at relevant events, not fixed intervals, and responds when the condition first becomes supported.Responses use only evidence observed by that time; trajectories may contain one or many responses.Source: A3. Causal dual-stream trajectory… · p. 23

A4. Response judging and full-trajectory revision (abridged)

  • Response judging checks factual correctness and trigger timing, removes repetitions or non-events, and identifies missed response opportunities.Revision regenerates the entire trajectory, correcting, removing, or adding responses rather than patching isolated entries.Source: A4. Response judging and… · p. 23

A5. Inner-state effectiveness test (abridged)

The inner-state effectiveness test reconstructs target responses using two contexts that differ in their access to visual history and hidden state.

A5. Inner-state effectiveness test (abridged)

  • For each target response, the test constructs a visual-reference context and an inner-state context.The visual-reference context uses the query, prior user-visible responses, and frames since the previous response, excluding hidden-state entries.Source: A5. Inner-state effectiveness test… · p. 24
  • The inner-state context includes the query, prior hidden-state and response entries, and only the most recent visual window.Earlier visual evidence is excluded from this context.Source: A5. Inner-state effectiveness test… · p. 24
  • A reconstruction must produce a short response supported at the end of its context; otherwise, it outputs exactly NO_RESPONSE_YET.Predictions are judged for correctness against the query; a missing key event, contradiction, different subject or action, refusal, empty output, or unwarranted NO_RESPONSE_YET is incorrect when the reference is substantive.Source: A5. Inner-state effectiveness test… · p. 24
  • A trajectory is retained only if its inner-state context reconstructs every applicable target response.The visual-reference result records whether removing visual history preserves capability or exposes information supplied by hidden state.Source: A5. Inner-state effectiveness test… · p. 24

F. Case Studies

Figure 7 illustrates SyncPro coordinating evidence tracking and requested outputs through two branches that can activate together or separately, with periods of silence.

F. Case Studies

  • Inner records relevant observations, while Response provides the outputs requested by the user.Figure 7 marks Inner updates in purple and Response outputs in pink.Source: F. Case Studies · p. 25, Figure 7 · p. 25
  • The two branches can activate together or separately, with periods of silence.Source: F. Case Studies · p. 25
  • Their independent activation coordinates ongoing evidence tracking with selective user-facing responses.Source: F. Case Studies · p. 25

Related papers

Loading 2610.32132v1…