Source-linked AI summary
EgoSIS: From Factorized Visual Ego-Transitions to Motion-Canonical Spatial Evidence for UAV Reasoning
Jingpu Yang, Fengxian Ji, Mingxuan Cui, Yilin Sun, Hang Zhang, Jianhua Zhu, Yufeng Wang
TL;DR
UAV video reasoning needs a stable spatial reference to distinguish camera motion from scene change. EgoSIS builds motion-canonical evidence from flow through transition factorization, reliability-gated memory, and local alignment. On SIS-Bench, it reaches 76.2% overall accuracy versus 63.1% for Qwen3-VL-8B-Instruct.
Problem
RGB-only multimodal models lack an explicit stable reference for separating camera motion from scene change, an ambiguity that matters for temporal spatial questions.
Source: §1 · p. 1Method
EgoSIS factorizes bidirectional flow, retains reliable transition history with re-anchoring, and aligns supported visual evidence to segment anchors without changing visual-token count.
Source: §1 · pp. 1–2, §2.1 · p. 2Results
EgoSIS-8B reaches 76.2% overall accuracy on SIS-Bench versus 63.1% for Qwen3-VL-8B-Instruct, with its largest improvements in action recognition and temporal recall.
Source: §3.2 · p. 4Takeaways & Limitations
EgoSIS provides a local motion-canonical reference while preserving Qwen’s visual-token count.
Source: §1 · p. 2, §4 · p. 4The representation is an image-plane proxy, not metric pose or a 3-D map.
Source: §1 · p. 2
Abstract
from arXiv · showhide
UAV video question answering requires separating camera motion from changes in the scene, but RGB-only multimodal models receive no explicit, stable reference for that separation. We present EgoSIS, a pose-free adapter that converts RGB-derived bidirectional flow into motion-canonical visual evidence in three stages. Factorized Visual Ego-Transitions (FVET) fits a robust image-plane transition and exposes motion, residual-support, and reliability factors. Reliability-Gated Ego-Transition Memory (ReTEM) uses reliability-weighted updates for a bounded history and re-anchors it at cuts or sustained uncertainty. Ego-Aligned Spatial Evidence (EASE) warps supported visual features into each segment's local anchor and injects four spatial evidence tokens per visual slice through zero-initialized residuals, without changing Qwen's visual-token count. On SIS-Bench, EgoSIS-8B obtains 89.9\% perception, 82.5\% perception-plus-memory, and 76.2\% overall accuracy, with the largest gains concentrated in self-awareness perception and memory. The adapter thus provides an interpretable interface between optical flow and spatial reasoning.
1. INTRODUCTION
UAV video reasoning must distinguish platform motion from scene change, but RGB-only models lack an explicit stable reference. EgoSIS factorizes visual transitions, retains reliable history, and aligns supported evidence to a local anchor.
1. INTRODUCTION
- Camera motion entangles platform movement with scene changes, while RGB-only MLLMs lack an explicit stable reference for temporal spatial questions.Apparent displacement can also reflect independently moving objects, occlusion, and artifacts.Source: §1 · p. 1
- Dense, view-dependent optical flow alone does not separate global from residual motion, identify unreliable correspondences, or define comparable coordinates across segments.The paper therefore uses a chain of transition factorization, reliable history retention, and alignment to a local anchor.Source: §1 · p. 1
- FVET factorizes bidirectional visual transitions into global image-plane motion, residual/static support, and reliability.ReTEM adds bounded transition memory with reliability gates, cut-aware segmentation, and safe geometric re-anchoring.Source: §1 · p. 2
- EASE converts reliable transition history into local motion-canonical evidence without expanding the visual-token sequence.The representation is an image-plane proxy, not metric pose or a 3-D map.Source: §1 · p. 2
2. EGOSIS
EGOSIS pairs frozen Qwen visual tokens with bidirectional-flow transition packets, retained segment history, and anchor-aligned evidence.
2.2. Factorized Visual Ego-Transitions
- FVET fits a robust affine transition from bidirectional flow and tokenizes translation, deformation, residual/support, and reliability factors.If too few consistent points survive, the edge remains time-aligned but contributes zero support and confidence.Source: §2.2 · p. 2, §2.1 · p. 2
2.1. Overview
- Contexts Z, M, and E update frozen Qwen visual features through zero-initialized residuals while preserving the visual-token count.The method combines FVET, ReTEM, and EASE contexts without changing Qwen’s visual-token count.Source: §2.1 · p. 2, Figure 2 · p. 3
2.3. Reliability-Gated Ego-Transition Memory
- ReTEM reliability-weights semantic updates, composes safe anchor-to-frame geometry, and re-anchors at cuts, persistent low confidence, unsafe candidates, quality drops, or the segment horizon.The memory bank retains a bounded history of past segments and combines it with current history to produce context M.Source: §2.3 · p. 3
2.4. Ego-Aligned Spatial Evidence
- EASE warps supported visual maps into segment anchors, excludes groups crossing cuts or segment boundaries, and pools valid evidence into four context tokens.A 2 × 2 pool yields the four internal context tokens E.Source: §2.4 · p. 3
3. EXPERIMENTS
The experiments evaluate EgoSIS on SIS-Bench and other UAV benchmarks, then examine how its three stages contribute. EgoSIS-8B leads the reported overall SIS-Bench result, while cross-benchmark performance and task-level gains vary.
3.1. Experimental Setup
- Structural variants use one training epoch on SIS-Motion-54K, with the Qwen backbone and flow estimator frozen and the connector and language-attention LoRA adapters trained.The formal stagewise run uses 54,298 training instances from 11,763 videos, sampled at 2 FPS and clamped to 8–32 frames; prompts, decoding, and score denominators are held fixed within comparisons.Source: §3.1 · p. 3
3.2. Main Results
- 76.2% overall accuracy gives EgoSIS-8B the highest Table 1 result, versus 63.1% for Qwen3-VL-8B-Instruct.The largest improvements over this backbone are in recognizing UAV actions and recalling temporal history; gains in reasoning are less consistent, and path planning remains the weakest task.Source: §3.2 · p. 4
3.3. Cross-Benchmark Generalization
- 97.3%/93.8%/95.6% across the reported OpenUAV-QA columns accompany leads in CameraBench classification (44.6%) and MotionBench CM (59.0%).EgoSIS-8B ties Ovis2.5-9B at 29.7% MotionBench Overall but trails it on CameraBench VQA, 53.6% versus 61.5%; comparisons with EgoSIS-7B do not isolate model scale because backbone generations differ.Source: Table 2 · p. 4, §3.3 · p. 4
3.4. Ablation and Analysis
- FRE reaches 76.2% overall accuracy versus 65.2% for visual-only SFT and performs best on six of seven tasks, while FR retains slightly higher spatial consistency.FVET improves action recognition and spatial consistency, ReTEM strengthens both but slightly reduces action recall and prediction, and EASE improves recall and prediction beyond F and FR; the comparisons include additional optimization at each stage.Source: §3.4 · p. 4, Table 3 · p. 4
4. CONCLUSION
EgoSIS provides a pose-free chain for factorizing flow, retaining reliable history, and aligning supported visual evidence. On SIS-Bench, it reaches 76.2% overall accuracy, while remaining an image-plane proxy rather than metric pose.
- EgoSIS links FVET transition factorization, ReTEM reliability-gated history and re-anchoring, and EASE alignment of supported evidence without changing visual-token count.Its spatial representation is an image-plane proxy, not metric pose or a 3-D map.Source: §4 · p. 4
- Because image displacement can conflate platform motion with scene change, UAV video reasoning needs a stable spatial reference; on SIS-Bench, EgoSIS-8B reaches 89.9% perception, 82.5% perception-plus-memory, and 76.2% overall accuracy.Source: §4 · p. 4
- Cross-benchmark results are mixed, and checkpoint-controlled progressive ablations are still needed to isolate ReTEM and EASE contributions.Source: §4 · p. 4