Moving Beyond More Views: Redundancy-Aware Ego-Exo Fusion for Proficiency Estimation

• Egocentric video captures subtle motion and intention cues, while exocentric views add full-body dynamics and spatial context.Source: §1 · pp. 1–2
• Adding one extra view can reduce accuracy by 1–2%, as noisy or duplicate views dilute useful cues and enlarged feature spaces encourage view-specific memorisation.The paper identifies multiview redundancy and feature overfitting as jointly contributing to performance degradation.Source: §1 · pp. 2–3, Figure 1 · p. 2
• AdaMVS selects and fuses informative exocentric views, while VIB-GB compresses non-essential signals and balances gradients to reduce overfitting.Together, the modules form a framework that jointly learns what to fuse and how to regularise fusion.Source: §1 · p. 3
• Experiments on EgoExo-4D and EgoExo-Fitness report state-of-the-art proficiency estimation with improved robustness and efficiency.Source: §1 · p. 3

Egocentric and Exocentric Understanding

• First-person views capture hand–object interactions and attention, while third-person views provide body motion and scene context; joint learning is challenged by viewpoint gaps and redundant information.Source: §2 · p. 3
• AE2 learns fine-grained view-invariant actions and LangView uses weakly supervised language-guided selection, but prior approaches leave uninformative-view noise and redundancy unresolved.Ego–Exo datasets support proficiency estimation, action anticipation, person localisation, and 3D pose estimation; scalability of cross-view alignment is constrained by limited paired data. The proposed method uses token-level selection to adaptively evaluate and fuse salient information.Source: §2 · pp. 3–4

Proficiency Estimation

• Existing proficiency-estimation methods primarily use late fusion, while SkillFormer achieves strong EgoExo-4D results; heterogeneous streams still pose redundancy and overfitting challenges.AQA predicts continuous quality scores, whereas proficiency estimation classifies discrete skill levels. The proposed approach uses adaptive token-level fusion to emphasise discriminative views and regularised training to enhance generalisation.Source: §2 · p. 4

Multi-view Learning

• Pixel/token fusion methods can be costly or miss dynamically varying view value, and views generalise at different rates; the paper combines adaptive selection and information-theoretic regularisation to minimise OGR.The framework is presented as jointly addressing feature redundancy and overfitting for generalisable Ego–Exo fusion.Source: §2 · pp. 4–5

3.1 Problem Formulation

• Proficiency estimation is formulated as discrete ordinal classification over synchronised ego–exo videos, rather than regression on continuous action-quality scores.The fusion function maps multiview video representations to one of K skill levels.Source: §3.1 · p. 5

3.2 AdaMVS

• AdaMVS ranks exocentric views by averaged token-importance scores and selects Top-K views, using shared queries and a permutation-invariant transformer to support varying view counts.Scores are learned weakly from task loss without view-level labels; Gumbel-Softmax weighting is used in training, while inference uses soft fusion to retain complementary cues. Ego and exo tokens are then adaptively weighted at token level.Source: §3.2 · p. 6

3.3 VIB-GB

• VIB-GB addresses fusion overfitting by balancing ego and exo gradients using OGR and compressing each branch’s latent representation with a variational information bottleneck.OGR-based weights are updated online to reduce gradients for branches that overfit faster; the VIB bottleneck limits input information while retaining task-relevant features, filtering view-specific noise.Source: §3.3 · pp. 7–8

3.4 Optimisation

• Training combines CORN ordinal loss, which decomposes proficiency levels into rank-consistent binary subtasks, with gradient-blending and VIB losses for robust, generalisable representations.The weighted multi-objective loss jointly promotes rank-aware prediction and mitigates redundant or overfit view signals.Source: §3.4 · pp. 8–9

4 Experiments

• The evaluation covers demonstrator proficiency on Ego-Exo4D and interpretable action judgments on EgoExo-Fitness, using one unified model per dataset and ego-only, exo-only, and ego+exo settings.Ego-Exo4D spans nine activity categories and has 2,987 proficiency scores; EgoExo-Fitness contains 1,276 cross-view videos segmented into 6,131 actions.Source: §4 · p. 9, §4.1 · p. 10

4.1 Comparison with State-of-the-Art

• AdaMVS achieves a new state-of-the-art on Ego-Exo4D, improving overall accuracy over SkillFormer by 5.5% absolute, and gains 7.1% absolute on EgoExo-Fitness; under matched backbones, gains are 12.7%, 3.1%, and 9.1%.The matched-backbone gains are for K400, K600, and HowTo100M, respectively.Source: §4.1 · p. 10

4.2 Comparison with General Fusion and Overfitting Baselines

• AdaMVS outperforms the compared fusion methods, while AdaMVS-Small retains similar accuracy at 0.26 GFLOPs and 2.19 M parameters, reducing computation and model size by over 11×.The paper also reports that VIB-GB outperforms generic overfitting-mitigation baselines by targeting branch-wise overfitting and balancing heterogeneous Ego–Exo learning dynamics.Source: Table 2 · p. 11, §4.2 · p. 11, Table 3 · p. 12

Effectiveness of Different Proposed Modules

• VIB-GB alone raises the naive baseline to 48.7% (+3.2%), but remains 2.3% below the full model, supporting the complementarity of data-level view filtering and feature-level regularisation.AdaMVS suppresses uninformative or redundant views, while VIB-GB reduces memorisation of view-specific noise.Source: Effectiveness of Different Proposed… · p. 12

4.3 Ablation Study

• 51.0% accuracy is the best reported Ego–Exo ablation result, rising from 45.5% through AdaMVS (48.0%), CORN loss (48.4%), and Gradient Blending (48.6%) before adding VIB.The reported gains are +2.5% for AdaMVS, +0.4% for CORN loss, and +2.4% for VIB added to Gradient Blending; the authors attribute the improvements to adaptive filtering, ordinal supervision, and latent-space regularisation.Source: §4.3 · p. 12, Table 4 · p. 13

Effectiveness of AdaMVS and View Selection

• AdaMVS uses weak supervision to select informative views; lower redundancy-index views receive higher weights, and predicted scores generally favor clear views over occluded or blurred ones.The redundancy index combines view-task relevance estimated by mutual information with inter-view similarity measured by CKA; similar scores in rare highly overlapping cases make selection difficult.Source: Effectiveness of AdaMVS and View… · p. 13, Table 5 · p. 13, Figure 5 · p. 15
• K = 2 yields the highest accuracy for exocentric view sampling; two dominant views account for approximately 82.8% of videos, while additional views can add redundant or noisy information.The paper notes that conventional fusion strategies often degrade as view count increases.Source: Effectiveness of Different Proposed… · p. 12, Effectiveness of AdaMVS and View… · pp. 13–14, Table 7 · p. 14

OGR and VIB-GB Analysis

• 53.0% accuracy with VIB-GB versus 48.7% for the baseline accompanies smoother training and weaker overfitting; its OGR decreases after an early rise and stabilises around epoch 20.Without VIB-GB, OGR steadily increases, indicating progressive overfitting.Source: Figure 4 · p. 14, OGR and VIB-GB Analysis · p. 14

Conclusion

• AdaMVS selects informative exocentric views at the data level, while VIB-GB combines a variational information bottleneck with gradient blending at the feature level to alleviate overfitting.Together, the modules address multiview redundancy and model overfitting in Ego–Exo proficiency estimation.Source: §5 · p. 15
• The complementary modules enhance system robustness and provide a general principle for adaptive fusion across heterogeneous ego-exo and homogeneous exo views.Their modular design also allows extension to other multiview and multimodal tasks.Source: §5 · p. 15
• Subjective proficiency annotations and limited dataset scale remain challenges, and future work will extend the approach to broader multiview tasks.The paper states that the framework demonstrates strong capability in mitigating these challenges.Source: §5 · p. 15

Additional Visualizations

• AdaMVS weights informative viewpoints highly and suppresses occluded, rear-facing, or background-dominated views, but assigns similar scores when all views are comparable.The latter cases reflect ambiguity when views have weak diversity or provide similarly informative signals.Source: A Additional Results · p. 20, Figure 1 · p. 21

View-wise Fusion Analysis

• Conventional fusion can decline as views increase, whereas AdaMVS maintains stable gains by selecting informative, complementary viewpoints for each action.Table 1 evaluates fusion methods across one to four exocentric views and ego–exo combinations.Source: A Additional Results · p. 20, Table 1 · p. 22

Redundancy/Overfitting Analysis

• 49.2 is achieved by AdaMVS through redundancy filtering and mitigation of overfitting.In the controlled study, adding identical views reduced performance, indicating branch-wise overfitting from increased capacity without new information; using all Exos also underperformed the selective approach.Source: A Additional Results · pp. 20, 22, Table 2 · p. 22

Ego + Single-exo-view Comparison

• The dominant exocentric view varies across methods, so no single fixed view suffices; AdaMVS provides task-agnostic robustness without manual tuning.The per-view comparison pairs the egocentric view with individual exocentric views.Source: A Additional Results · p. 22, Table 3 · p. 22

B Metrics for Quantifying Multiview Redundancy

• Multiview redundancy arises when a view contributes little to proficiency prediction or repeats information already encoded by other views.These are the complementary factors of low task informativeness and high representational overlap.Source: B Metrics for Quantifying Multiview… · p. 23
• MI measures each view’s task relevance, while CKA captures representation-level similarity across views.Together, they quantify redundancy through informativeness and inter-view overlap.Source: B Metrics for Quantifying Multiview… · p. 23
• For MI, each view’s feature embedding is evaluated against the ordinal proficiency score; higher MI indicates stronger task-relevant cues, while low MI suggests a noisy or uninformative view.MI measures the statistical dependency between an exocentric view and the proficiency label.Source: B Metrics for Quantifying Multiview… · p. 23

Centred Kernel Alignment (CKA)

• Linear CKA quantifies similarity between feature representations from two views, ranges from 0 for dissimilar to 1 for identical up to rotation, and is invariant to isotropic scaling and orthogonal transformations.High CKA indicates overlapping or redundant information, making it useful for diagnosing redundancy among neural representations.Source: Centred Kernel Alignment (CKA) · p. 23
• The redundancy index combines a view’s mean CKA similarity to other views with its mutual information; lower values indicate more informative, less redundant views.A small ϵ avoids division by zero.Source: Centred Kernel Alignment (CKA) · p. 23
• Views with lower redundancy indices consistently receive higher AdaMVS importance weights, showing that AdaMVS prioritizes informative, complementary views.The observed correspondence aligns learned importance with redundancy patterns in the data.Source: Centred Kernel Alignment (CKA) · p. 24

B.1 Overfitting-to-Generalisation Ratio (OGR)

• A higher OGR means optimisation is dominated more by overfitting than by genuine generalisation improvement.OGR quantifies how much overfitting increases relative to the improvement in generalisation.Source: §B.1 · p. 24
• Gradient Blending adaptively combines modality gradients, assigning larger weights to gradients that generalise well and smaller weights to those prone to overfitting.Gradient weights reflect each modality’s generalisation contribution relative to its overfitting variance.Source: §B.1 · p. 24
• Across Ego–Exo view configurations, highly correlated validation losses produce nearly identical generalisation-gain trajectories, limiting their ability to distinguish view-specific generalisation.Validation-loss fluctuations also make the generalisation-gain term unstable.Source: §B.1 · p. 24
• The method therefore uses a simplified OGR focused on overfitting increments, which separate ego and exo branches as ego features typically converge faster and overfit earlier.This choice makes gradient reweighting respond directly to the view that overfits more rapidly and improves training stability.Source: §B.1 · p. 25

C.2 AdaMVS Pseudocode

• AdaMVS selects informative exocentric views and adaptively weights egocentric and exocentric tokens to generate a robust final embedding.Its exocentric procedure scores tokens, applies Gumbel–Softmax weights, and uses Top-K view masking during training; test-time fusion is soft and omits Top-K.Source: §C.2 · p. 25, D Extended Related Work · p. 27

C.1 Transformer Decoder Structure

• Learnable queries use self-attention and feed-forward layers to refine representations before attending to encoded features through multi-head attention.Residual connections and layer normalisation follow each sub-layer to support stable optimisation.Source: §C.1 · p. 25, Figure 3 · p. 26

D Extended Related Work

• Instructional-video and sports skill-analysis methods often use temporally aggregated features and static fusion, whereas AdaMVS dynamically weights each view and temporal token.This token-level weighting is intended to control redundant or noisy visual cues during skill evaluation.Source: D Extended Related Work · p. 26
• Traditional multimodal fusion uses fixed early, intermediate, or late strategies, while transformer methods enable token-wise interaction but can have scalability limits in asynchronous ego–exo settings.The framework draws on fine-grained fusion while adding view adaptivity and view-level regularisation to control redundancy and overfitting.Source: D Extended Related Work · p. 27
• Prior work reports that views can overfit or underfit at different rates, but few methods explicitly optimise this generalisation imbalance.VIB-GB is presented as complementing AdaMVS by suppressing view-specific overfitting and balancing learning between ego and exo streams.Source: D Extended Related Work · p. 27

E Limitations and Future Work

• Weakly supervised view-importance scores derive from subjective, sometimes ambiguous proficiency annotations, which may reduce precision when annotators disagree or skill cues are subtle.Source: E Limitations and Future Work · p. 27
• Adaptive Top-K can mitigate but cannot fully resolve cases where multiple views contribute similarly.Source: E Limitations and Future Work · p. 27

1. Feature projection

• The feature-projection step projects both modalities before fusion.Source: §1 · p. 28

Adaptive weighting

• AdaMVS applies Gumbel–Softmax weights, broadcasts them to tokens, and softly fuses both modalities into features for downstream decoding.The returned fused features are denoted xfinal.Source: §3 · p. 28
• For simple motions or highly overlapping camera coverage, near-equivalent views can receive almost uniform weights, preventing AdaMVS from reliably identifying a dominant view.The passage describes this as an inherent limitation of the data rather than the method.Source: §3 · p. 28
• Future work will explore dynamically selecting the number of views based on action complexity and cross-view variability, and extend proficiency estimation to language-based feedback generation.The proposed extension integrates Ego–Exo action understanding with language-based feedback for more comprehensive and interpretable performance analysis.Source: §3 · p. 28