LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistant

Motivation

• Online video assistants must continuously receive first-person streams and provide real-time, proactive responses or guidance.These requirements impose high demands on both efficacy and efficiency.Source: §1 · p. 1

Limitations of Existing Methods

• Existing offline video understanding methods are unsuitable for online video assistants, while LIVE processes low-frame-rate video with coarse visual features.These constraints hinder temporal understanding and response accuracy.Source: §1 · p. 1
• LIVE retains limited tokens across frames and does not exploit first-person visual characteristics, producing imprecise responses and inefficient token expansion.The approach expands tokens across all frames even though substantial expansion is mainly needed for keyframes during response generation.Source: §1 · p. 2

LION-FS

• LION-FS combines general spatial features, dense temporal features, and first-person features through Token Aggregation Routing without increasing token numbers.Token Dropping Routing further removes redundant features to improve response determination efficiency.Source: §1 · p. 2
• LION-FS uses fast thinking for simple response determination and slow thinking for complex response generation through a two-path framework combining fine-tuning and training-free methods.Evaluations on Ego4D and Ego-Exo4D report optimal performance and efficiency on online first-person video dialogue tasks.Source: §1 · p. 2

Online Visual Dialogue

• Offline visual understanding commonly supports complete-image or complete-video input to multimodal language models, while some work extends dialogue to interleaved images and continuous video streams.These lines of research provide context for online visual dialogue.Source: §2 · p. 2

The Fast and Slow Concept

• The fast-and-slow concept separates intuitive processing from deliberate reasoning, and prior video methods use different frame rates or selective activation for complex visual queries.LION-FS adopts this conceptual distinction for online video assistance.Source: §2 · p. 2

Routing-Based Modeling

• Routing-based modeling selects experts or tokens for computation, with mixture-of-experts and mixture-of-depths methods reducing unnecessary multimodal processing.LION-FS adapts these routing ideas to aggregate diverse visual tokens and discard redundant ones in online video dialogue.Source: §2 · pp. 2–3

LION-FS Framework

• LION-FS’s framework combines a Fast Path for high-frame-rate response determination with a Slow Path that augments keyframes using Grid Tokens and Box Tokens.The augmented features are inserted into a multimodal thinking template to guide response generation.Source: Figure 2 · p. 3

Overall framework

• The framework separates online video dialogue into Fast Path response determination and Slow Path response generation.The Fast Path processes incoming frames, while the Slow Path generates an answer when a response is warranted.Source: §3 · p. 3, §3.1 · p. 3

Fast Path

• Dual encoding combines general spatial features with first-person temporal features for higher-rate video processing.EgoVLPv2 processes grouped 8 FPS frames, while SigLIP processes frames at 2 FPS, producing complementary visual sequences.Source: §3.2 · p. 4
• The Token Aggregation Router adaptively fuses general and first-person visual features without increasing output token numbers.The router uses adaptive weights generated from visual guidance to combine spatial and dense temporal information.Source: §3.2 · p. 4, Figure 3 · p. 4
• The Token Dropping Router retains only high-weight visual tokens to remove redundancy from interaction regions and similar consecutive frames.The method selectively discards redundant tokens according to routing weights and a user-defined discard proportion.Source: §3.2 · pp. 4–5
• The Routing-Based Fast Path achieves a fourfold frame-rate increase over LIVE while enriching temporal information for response determination.It is designed to improve real-time processing and response-determination accuracy through adaptive feature routing.Source: §3.2 · p. 5

Slow Path

• The training-free Slow Path augments response-triggering keyframes with global Grid tokens and local Box tokens for finer visual representation.Uniform grids preserve regional spatial details, while augmented keyframes are integrated into the interleaved frame-text dialogue.Source: §3.3 · p. 5

Main Results

• LION-FS outperforms existing video-streaming dialogue methods on most metrics across the Ego4D and Ego-Exo4D narration-stream benchmarks.The evaluation uses LM-PPL, LM-Correctness, TimeDiff, and Fluency to assess language modeling and temporal alignment.Source: Table 1 · p. 6, §4.2 · p. 6, §4.1 · p. 6

Analysis on Token Aggregation Router

• Adaptive routing aggregation improves visual integration and captures temporal-spatial correlations without increasing the fused token count.The aggregation ablation compares SigLIP and EgoVLPv2 features; different encoders complement each other across LL-PPL, TimeDiff, Fluency, and LM-Correctness.Source: Table 2 · p. 6, §4.3.1 · pp. 6–7

Main Results

• 4× higher frame-rate processing than prior methods demonstrates LION-FS’s efficiency advantage in online video dialogue.The paper reports a slight TimeDiff disadvantage on Ego4D Narration Validation, attributed to a short 6.73-word average response length, while all metrics improve on Ego-Exo4D Narration Validation with 10.96-word responses.Source: §4.2 · p. 6

Analysis on Token Dropping Router

• Interleaved Layers with β = 0.5 provides a balance between model performance and efficiency for the token-dropping router.Random dropping degrades performance, whereas the learnable router can retain satisfactory efficacy with fewer visual tokens.Source: Table 3 · p. 7, §4.3.2 · p. 7

Analysis on Token Aggregation Router

• The SigLIP CLS token most effectively optimizes weights between visual tokens because of its rich knowledge and generalization ability.Figure 4 visualizes token-aggregation routing outcomes using Egen weights, with Token 1 denoting the CLS token.Source: Table 4 · p. 7, Figure 4 · p. 7

Analysis on Multi-granularity Augmentation

• Combining Grid Tokens and Box Tokens enriches keyframe visual information and improves language-modeling capability on LL-PPL and LM-Correctness.The 4×3×3 Grid Tokens pattern aligns keyframes with the training mode, while Local Adaptive Augmentation prompts further improve capability; qualitative results also show more precise and detailed responses.Source: §4.3.3 · p. 8, Table 5 · p. 8

• LION-FS addresses online video assistants’ efficacy and efficiency challenges with a Fast & Slow optimization scheme.Source: §5 · p. 8
• The Fast Path dynamically integrates general image and first-person video features with Token Aggregation and removes redundancies using Token Dropping.Source: §5 · p. 8
• The Slow Path improves response precision through global uniform and local adaptive keyframe augmentation focused on human-environment interactions.Source: §5 · p. 8

• SigLIP and EgoVLPv2 provide complementary visual features that are aligned to text-token dimensions through MLP processing.SigLIP uses 2-FPS image frames, while EgoVLPv2 processes 8-FPS frames in groups for temporally aligned video features.Source: §A.1 · p. 9, §A.2 · p. 9
• The Token Aggregation Router assigns frame-level weights to fuse visual tokens, while the Token Dropping Router removes tokens below a confidence threshold.Source: §A.1 · p. 9
• Ego-Exo4D annotations are rewritten into second-person descriptions while preserving capitalization to distinguish other individuals.Source: §A.2 · p. 9
• Dialogue data augmentation introduces misinformation, temporal inconsistencies, and missing messages to train responses under varied online conditions.Source: §A.2 · p. 9

• Training runs on 8 × A800 80GB GPUs with batch size 1 per GPU and gradient accumulation over 32 steps.Source: §A.3 · p. 9
• The model is trained for 10 epochs on Ego-Exo4D and 2 epochs on Ego4D.Source: §A.3 · p. 9
• Optimization uses AdamW with an initial learning rate of 0.0002, cosine scheduling, and a 5% warmup ratio.Source: §A.3 · p. 9

• LM-PPL evaluates probabilistic language modeling quality, with lower values typically indicating stronger language modeling capability.Source: §A.4 · p. 9
• LM-Correctness measures the proportion of correctly generated tokens against reference text.Source: §A.4 · p. 9
• TimeDiff measures temporal alignment as the difference between response and expected timestamps, averaged across dialogue turns.Source: §A.4 · pp. 9–10
• Fluency measures naturalness and coherence through successfully predicted tokens across response determination and generation.Source: §A.4 · p. 10

Routing and keyframe efficiency

• The Fast Path aggregates visual tokens and drops redundant ones, while the Slow Path augments only responsive keyframes to limit efficiency impact.The two routers support high-frame-rate processing, and ordinary silent frames avoid augmentation.Source: §B.1 · p. 10
• LION-FS uses customized routers and keyframe augmentation to reduce FLOPs at 8 FPS, while avoiding frame augmentation keeps FLOPs nearly constant.The comparison uses the same Ego-Exo4D test sample and a dropping ratio of β = 0.5 for both compared systems.Source: Table 6 · p. 10

Interaction-aware features

• Interaction regions occupy a small part of first-person frames, which can disperse MLLM attention across the scene.The motivation focuses on user-environment interaction areas in Ego-Exo4D and Ego4D videos.Source: §B.2 · p. 10
• Box Tokens pool each detected interaction box into one representation for up to two hands and one interacting object, substituting a global frame token when detections are absent.When interaction regions are absent, the method emphasizes the global scene instead.Source: §B.2 · p. 11

Training constraints

• The response-generation labels are long-tailed because [EOS] occurs much more frequently than [Assistant], encouraging silence predictions during training.The proposed remedies include multiclass response-probability prediction or response-probability regression.Source: D. Limitations · p. 11

Deployment risks

• LION-FS may produce hallucinated or biased responses, and unreliable dataset annotations motivate critical evaluation and safety and fairness assessments before deployment.The model is fine-tuned on Ego-Exo4D and Ego4D, whose annotation reliability is identified as a deployment concern.Source: E. Societal Impact and Potential Risk · p. 11