Fovea-JEPA: Overcoming Geometric Blind Spots in V-JEPA for Camera Motion Recognition

1 Introduction

• Camera-motion recognition must separate translation, rotation, and zoom from object motion and scene dynamics, which semantic correspondences alone may not capture reliably.Different camera operations can look similar, and independently moving objects can resemble camera-induced motion.Source: §1 · p. 1
• Under temporal-reversal supervision, V-JEPA improves overall but declines for several motion types, whereas the weaker VGGT predictor improves across every evaluated motion class.The contrasting responses suggest VGGT retains complementary geometric cues where V-JEPA is less consistent.Source: §1 · p. 1
• Fovea-JEPA uses a token-wise map Q and clip-level score q to selectively deploy VGGT; matched-budget controls favor high-q over low-q or random intervention.Indiscriminate fusion risks interfering with reliable V-JEPA predictions.Source: §1 · pp. 1–2, Figure 1 · p. 2
• The paper proposes adaptive VGGT-based geometric intervention and weakly supervises a predictor that localizes geometrically salient regions with token-level response maps.V-JEPA remains the primary representation, while VGGT supplies complementary geometric information.Source: §1 · p. 2

2.1 Camera Motion Understanding

• CameraBench, ShotBench, and CineTechBench (Wang et al. 2025b) evaluate camera and cinematic tasks; VGGT predicts camera, depth, point maps, and tracks.Geometry-guided camera-motion methods also inject 3D-derived cues into VideoLLMs (Feng, Musunuri, and Su 2026).Source: §2.1 · p. 2

2.2 Video Representation Learning

• Video representation work spans contrastive and masked prediction, while temporal-order objectives learn motion-sensitive structure; this paper uses temporal reversal to compare V-JEPA and VGGT across camera-motion families.Prior temporal-order approaches include forward–backward classification, label reversal, and time-equivariant learning (Pickup et al. 2014; Price and Damen 2019; Jenni and Jin 2021; Wei et al. 2018; Jenni, Meishvili, and Favaro 2020).Source: §2.2 · pp. 2–3

2.3 Joint-Embedding Predictive Architectures

• I-JEPA’s masked latent prediction extends to video in V-JEPA and scales through V-JEPA 2 and 2.1, while fine-grained geometric limits remain underexplored (Assran et al. 2025; Mur-Labadia et al. 2026).A systematic study reports strengths in corruption robustness, physical discrimination, and temporal-direction sensitivity (Alrasheed et al. 2026).Source: §2.3 · p. 3

3.1 Overview

• Fovea-JEPA maps video clips to camera-motion logits through a dual-branch V-JEPA/VGGT architecture whose fused context is modulated by a blind-spot score.The visual and geometric branches produce context tokens; the predictor uses their context and dense VGGT tokens to estimate where geometric intervention is useful.Source: §3.1 · p. 3, §3.2 · p. 4

3.2 Error-guided Intervention Labels

• Intervention labels identify clips where visual predictions are wrong but visual-geometric fusion is correct, while clips wrong in both branches receive no supervision.Correct visual-only predictions are labeled as not needing geometric modulation.Source: §3.2 · p. 4
• Token-level geometric saliency uses first-order gradient cues against a frame-averaged token baseline to localize camera-informative regions.High positive saliency indicates tokens whose absence would degrade prediction performance; video-level labels screen clips while token-level signals localize salient regions.Source: §3.2 · p. 4

3.3 Blind-Spot Map Estimation

• The blind-spot predictor combines cross-branch context with projected dense geometric tokens to produce a sigmoid-normalized spatiotemporal response map.It is trained using video-level intervention labels and geometrically salient maps.Source: §3.3 · pp. 4–5

3.4 Blind-Spot-Guided Geometric Intervention

• The response map guides token-level feature fusion and is pooled into a clip-level score that controls global modulation, filtering noisy responses before threshold-based selection.The visual prediction is retained when estimated geometric benefit is low; otherwise, the intervened prediction is selected.Source: §3.4 · p. 5

3.5 Optimization Objectives

• Training combines listwise, cosine, and pairwise map losses with clip-level intervention and preservation losses and the camera-motion classification loss.The intervention loss compares the clip score with a pseudo label, while the preservation loss minimizes the score when visual predictions are already correct.Source: §3.5 · p. 5

4.1 Experimental Setup

• Evaluation covers CameraMotionDataset, CameraBench, and ShotBench, using published general-purpose VLM and task-specific baselines and frozen V-JEPA 2.1 and VGGT backbones.The experiments uniformly sample eight frames per clip and use five-fold cross-validation with out-of-fold predictions for intervention labels.Source: §4.1 · p. 5

4.2 Main Results

• 8.67% mAP and 11.77% AB-Acc over Cam-Motion-72B on CameraBench, plus 24.38% over ShotVL-7B on ShotBench; Fovea-JEPA outperforms all baselines across three datasets.On CameraMotionDataset, it achieves 83.47% Inst. Acc., 93.47% Macro-F1, and 94.50% Weighted-F1, versus 73.80%, 87.00%, and 92.00% for the strongest baseline, the VGGT camera-cue classifier.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 Ablation Studies

• Temporal-reversal supervision improves overall exact-match accuracy for both branches, but V-JEPA 2.1 loses accuracy on tilt and truck while VGGT improves across every evaluated family.V-JEPA 2.1 pair accuracy rises 4.00 and 1.16 percentage points for roll and dolly, but falls 3.76 and 3.13 points for tilt and truck; VGGT remains the weaker standalone predictor.Source: Temporal Reversal and Geometric Blind… · p. 6, Table 2 · p. 6
• With a matched 50% token budget, top-ranked Q locations yield 0.8143 exact-match accuracy, versus 0.7622 for bottom-ranked locations.Random masks fall between the two ranked controls, indicating that high-Q locations contain geometric evidence facilitating camera-motion prediction.Source: Temporal Reversal and Geometric Blind… · p. 6
• Clip-level q routing achieves 0.855 AUROC and 0.361 AUPRC against 0.099 rescue prevalence; high-q intervention reaches 0.8347 accuracy and +42 net corrections.At the same intervention budget, low-q routing yields 0.7761 accuracy and random routing 0.7937; descending-q routing consistently outperforms ascending-q and random routing across budgets.Source: Temporal Reversal and Geometric Blind… · p. 6, Figure 4 · p. 7
• In paired same-label examples, high-q intervention recovers exact labels when both branches are inexact, while low-q routing preserves V-JEPA 2.1’s exact prediction.The Q-maps localize relevant evidence across locations and frames, while q controls whether intervention is applied.Source: Figure 3 · p. 7, Temporal Reversal and Geometric Blind… · p. 7
• Removing action loss prevents any sample from crossing the intervention threshold, removing keep loss increases harmful correct-to-wrong transitions, and removing map-supervision terms degrades performance.Removing classification loss substantially decreases exact-match accuracy and net correction.Source: Temporal Reversal and Geometric Blind… · p. 7, Table 3 · p. 7

• Fovea-JEPA uses hierarchical pseudo-supervision with clip-level intervention labels and token-level geometric saliency targets to train a blind-spot predictor.Source: §5 · p. 7
• The predictor locates critical geometric regions and unreliable visual predictions, enabling adaptive token-level feature modulation and clip-level threshold-based prediction correction.Source: §5 · p. 7
• Extensive experiments validate the design, which combines robust visual generalization with precise geometric structural priors.Source: §5 · p. 7