Source-linked AI summary
LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistant
Wei Li, Bing Hu, Rui Shao, Leyang Shen, Liqiang Nie
TL;DR
Online first-person video assistants must balance real-time processing with accurate, detailed responses, but existing methods rely on low-frame-rate, coarse visual features. LION-FS uses routed fast-path processing and slow-path keyframe augmentation to address this trade-off, achieving strong efficacy and efficiency across online video dialogue benchmarks.
Problem
Online video assistants must continuously process first-person video streams while determining response timing and generating precise responses under real-time efficiency demands.
Method
LION-FS decouples response determination and generation through a Fast Path that routes spatial-temporal features and a Slow Path that augments keyframes with multi-granularity features.
Results
LION-FS consistently outperforms existing methods on online video dialogue evaluations and processes video streams at four times their frame rate.
Takeaways & Limitations
The two-path optimization scheme jointly improves online video assistant efficacy and efficiency across the reported first-person dialogue benchmarks.
Takeaways & Limitations
The response-generation decision uses a signaling token such as EOS, and the authors identify long-tailed token frequencies as a training challenge that can bias the model toward silence.
Abstract
from arXiv · showhide
First-person video assistants are highly anticipated to enhance our daily lives through online video dialogue. However, existing online video assistants often sacrifice assistant efficacy for real-time efficiency by processing low-frame-rate videos with coarse-grained visual features.To overcome the trade-off between efficacy and efficiency, we propose "Fast & Slow Video-Language Thinker" as an onLIne videO assistaNt, LION-FS, achieving real-time, proactive, temporally accurate, and contextually precise responses. LION-FS adopts a two-stage optimization strategy: 1)Fast Path: Routing-Based Response Determination evaluates frame-by-frame whether an immediate response is necessary. To enhance response determination accuracy and handle higher frame-rate inputs efficiently, we employ Token Aggregation Routing to dynamically fuse spatiotemporal features without increasing token numbers, while utilizing Token Dropping Routing to eliminate redundant features. 2)Slow Path: Multi-granularity Keyframe Augmentation optimizes keyframes during response generation. To provide comprehensive and detailed responses beyond atomic actions constrained by training data, fine-grained spatial features and human-environment interaction features are extracted through multi-granular pooling. These features are further integrated into a meticulously designed multimodal Thinking Template to guide more precise response generation. Comprehensive evaluations on online video tasks demonstrate that LION-FS achieves state-of-the-art efficacy and efficiency.
1. Introduction
Online first-person video assistants must continuously process video while responding to queries in real time and proactively, creating a demanding efficacy–efficiency trade-off. LION-FS addresses limitations in existing online methods with a Fast Path for response determination and a Slow Path for precise response generation.
- Motivation: Online video assistants must continuously receive first-person streams and provide real-time, proactive responses or guidance.These requirements impose high demands on both efficacy and efficiency.
- Limitations of Existing Methods: Existing offline video understanding methods are unsuitable for online video assistants, while LIVE processes low-frame-rate video with coarse visual features.These constraints hinder temporal understanding and response accuracy.
- Limitations of Existing Methods: LIVE retains limited tokens across frames and does not exploit first-person visual characteristics, producing imprecise responses and inefficient token expansion.The approach expands tokens across all frames even though substantial expansion is mainly needed for keyframes during response generation.
- LION-FS: LION-FS combines general spatial features, dense temporal features, and first-person features through Token Aggregation Routing without increasing token numbers.Token Dropping Routing further removes redundant features to improve response determination efficiency.
- LION-FS: LION-FS uses fast thinking for simple response determination and slow thinking for complex response generation through a two-path framework combining fine-tuning and training-free methods.Evaluations on Ego4D and Ego-Exo4D report optimal performance and efficiency on online first-person video dialogue tasks.
2. Related Works
Related work spans offline visual understanding, fast-and-slow video reasoning, and routing-based modeling. LION-FS applies these ideas to online video dialogue by routing diverse visual tokens and discarding redundant ones.
- Online Visual Dialogue: Offline visual understanding commonly supports complete-image or complete-video input to multimodal language models, while some work extends dialogue to interleaved images and continuous video streams.These lines of research provide context for online visual dialogue.
- The Fast and Slow Concept: The fast-and-slow concept separates intuitive processing from deliberate reasoning, and prior video methods use different frame rates or selective activation for complex visual queries.LION-FS adopts this conceptual distinction for online video assistance.
- Routing-Based Modeling: Routing-based modeling selects experts or tokens for computation, with mixture-of-experts and mixture-of-depths methods reducing unnecessary multimodal processing.LION-FS adapts these routing ideas to aggregate diverse visual tokens and discard redundant ones in online video dialogue.
- LION-FS Framework: LION-FS’s framework combines a Fast Path for high-frame-rate response determination with a Slow Path that augments keyframes using Grid Tokens and Box Tokens.The augmented features are inserted into a multimodal thinking template to guide response generation.
3. LION-FS
LION-FS decouples online video dialogue into a Fast Path for response determination and a Slow Path for response generation. It combines dual visual encoding, adaptive token routing, token dropping, and keyframe augmentation to improve real-time efficiency and response precision.
- Overall framework: The framework separates online video dialogue into Fast Path response determination and Slow Path response generation.The Fast Path processes incoming frames, while the Slow Path generates an answer when a response is warranted.
- Fast Path: Dual encoding combines general spatial features with first-person temporal features for higher-rate video processing.EgoVLPv2 processes grouped 8 FPS frames, while SigLIP processes frames at 2 FPS, producing complementary visual sequences.
- Fast Path: The Token Aggregation Router adaptively fuses general and first-person visual features without increasing output token numbers.The router uses adaptive weights generated from visual guidance to combine spatial and dense temporal information.
- Fast Path: The Token Dropping Router retains only high-weight visual tokens to remove redundancy from interaction regions and similar consecutive frames.The method selectively discards redundant tokens according to routing weights and a user-defined discard proportion.
- Fast Path: The Routing-Based Fast Path achieves a fourfold frame-rate increase over LIVE while enriching temporal information for response determination.It is designed to improve real-time processing and response-determination accuracy through adaptive feature routing.
- Slow Path: The training-free Slow Path augments response-triggering keyframes with global Grid tokens and local Box tokens for finer visual representation.Uniform grids preserve regional spatial details, while augmented keyframes are integrated into the interleaved frame-text dialogue.
4. Experience
LION-FS is evaluated on online egocentric video benchmarks using language-modeling and temporal-alignment metrics, with ablations examining routing and keyframe augmentation. Across these evaluations, it improves online video dialogue efficacy while supporting higher-frame-rate processing.
- Main Results: LION-FS outperforms existing video-streaming dialogue methods on most metrics across the Ego4D and Ego-Exo4D narration-stream benchmarks.The evaluation uses LM-PPL, LM-Correctness, TimeDiff, and Fluency to assess language modeling and temporal alignment.
- Analysis on Token Aggregation Router: Adaptive routing aggregation improves visual integration and captures temporal-spatial correlations without increasing the fused token count.The aggregation ablation compares SigLIP and EgoVLPv2 features; different encoders complement each other across LL-PPL, TimeDiff, Fluency, and LM-Correctness.
- Main Results: 4× higher frame-rate processing than prior methods demonstrates LION-FS’s efficiency advantage in online video dialogue.The paper reports a slight TimeDiff disadvantage on Ego4D Narration Validation, attributed to a short 6.73-word average response length, while all metrics improve on Ego-Exo4D Narration Validation with 10.96-word responses.
- Analysis on Token Dropping Router: Interleaved Layers with β = 0.5 provides a balance between model performance and efficiency for the token-dropping router.Random dropping degrades performance, whereas the learnable router can retain satisfactory efficacy with fewer visual tokens.
- Analysis on Token Aggregation Router: The SigLIP CLS token most effectively optimizes weights between visual tokens because of its rich knowledge and generalization ability.Figure 4 visualizes token-aggregation routing outcomes using Egen weights, with Token 1 denoting the CLS token.
- Analysis on Multi-granularity Augmentation: Combining Grid Tokens and Box Tokens enriches keyframe visual information and improves language-modeling capability on LL-PPL and LM-Correctness.The 4×3×3 Grid Tokens pattern aligns keyframes with the training mode, while Local Adaptive Augmentation prompts further improve capability; qualitative results also show more precise and detailed responses.
5. Conclusion
LION-FS is an online video-assistant framework that separates intuitive response determination from deliberative response generation. Its Fast and Slow paths jointly improve efficacy and efficiency through routing and multi-granularity keyframe augmentation.
- LION-FS addresses online video assistants’ efficacy and efficiency challenges with a Fast & Slow optimization scheme.
- The Fast Path dynamically integrates general image and first-person video features with Token Aggregation and removes redundancies using Token Dropping.
- The Slow Path improves response precision through global uniform and local adaptive keyframe augmentation focused on human-environment interactions.
A.1. Architecture
The architecture aligns visual features from complementary encoders and routes them through token aggregation and dropping. Training data is additionally refined and augmented to support robust online dialogue.
- SigLIP and EgoVLPv2 provide complementary visual features that are aligned to text-token dimensions through MLP processing.SigLIP uses 2-FPS image frames, while EgoVLPv2 processes 8-FPS frames in groups for temporally aligned video features.
- The Token Aggregation Router assigns frame-level weights to fuse visual tokens, while the Token Dropping Router removes tokens below a confidence threshold.
- Ego-Exo4D annotations are rewritten into second-person descriptions while preserving capitalization to distinguish other individuals.
- Dialogue data augmentation introduces misinformation, temporal inconsistencies, and missing messages to train responses under varied online conditions.
A.3. Training Settings
Training uses full MLP, routing modules, and LoRA components across eight A800 GPUs, with dataset-specific epochs and gradient accumulation. Optimization uses AdamW with cosine scheduling and warmup.
- Training runs on 8 × A800 80GB GPUs with batch size 1 per GPU and gradient accumulation over 32 steps.
- The model is trained for 10 epochs on Ego-Exo4D and 2 epochs on Ego4D.
- Optimization uses AdamW with an initial learning rate of 0.0002, cosine scheduling, and a 5% warmup ratio.
A.4. Evaluation Metrics
Evaluation measures language quality, response correctness, real-time temporal alignment, and dialogue fluency for online video dialogue. These metrics cover both generated content and timing behavior.
- LM-PPL evaluates probabilistic language modeling quality, with lower values typically indicating stronger language modeling capability.
- LM-Correctness measures the proportion of correctly generated tokens against reference text.
- TimeDiff measures temporal alignment as the difference between response and expected timestamps, averaged across dialogue turns.
- Fluency measures naturalness and coherence through successfully predicted tokens across response determination and generation.
B.1. Additional Efficiency Evaluation
LION-FS improves online video efficiency by routing visual tokens and selectively augmenting responsive keyframes. Its interaction-aware box tokens focus processing on hands and objects while preserving global-scene information when interactions are absent.
- Routing and keyframe efficiency: The Fast Path aggregates visual tokens and drops redundant ones, while the Slow Path augments only responsive keyframes to limit efficiency impact.The two routers support high-frame-rate processing, and ordinary silent frames avoid augmentation.
- Routing and keyframe efficiency: LION-FS uses customized routers and keyframe augmentation to reduce FLOPs at 8 FPS, while avoiding frame augmentation keeps FLOPs nearly constant.The comparison uses the same Ego-Exo4D test sample and a dropping ratio of β = 0.5 for both compared systems.
- Interaction-aware features: Interaction regions occupy a small part of first-person frames, which can disperse MLLM attention across the scene.The motivation focuses on user-environment interaction areas in Ego-Exo4D and Ego4D videos.
- Interaction-aware features: Box Tokens pool each detected interaction box into one representation for up to two hands and one interacting object, substituting a global frame token when detections are absent.When interaction regions are absent, the method emphasizes the global scene instead.
D. Limitations
The online video dialogue setting faces training and deployment constraints. Long-tailed response labels can bias training toward silence, while hallucinations, biased outputs, and unreliable annotations require caution before deployment.
- Training constraints: The response-generation labels are long-tailed because [EOS] occurs much more frequently than [Assistant], encouraging silence predictions during training.The proposed remedies include multiclass response-probability prediction or response-probability regression.
- Deployment risks: LION-FS may produce hallucinated or biased responses, and unreliable dataset annotations motivate critical evaluation and safety and fairness assessments before deployment.The model is fine-tuned on Ego-Exo4D and Ego4D, whose annotation reliability is identified as a deployment concern.