Source-linked AI summary
Multi-View Learning in the Presence of View Disagreement
C. Christoudias, Raquel Urtasun, Trevor Darrell
TL;DR
When views disagree because of corruption, noise, or occlusion, conventional multi-view methods can perform poorly. This paper uses conditional entropy to detect disagreement and reports successful learning on realistic noisy data, including audio-visual user-agreement learning at 50–70% disagreement.
Problem
Conventional multi-view methods assume views share a class and can perform poorly when substantial view disagreement violates that assumption.
Source: §2 · p. 2Method
The approach uses conditional entropy over joint view statistics to detect disagreement and filter affected samples before multi-view learning.
Source: §3 · pp. 3–4Results
Takeaways & Limitations
On noisy data, the approach exploits distinct foreground and background co-occurrence patterns to detect view disagreement reliably.
Source: §6 · p. 8The method assumes foreground samples co-occur only with their class or background, while background samples can co-occur with any class.
Source: §3 · p. 3
Abstract
from arXiv · showhide
Traditional multi-view learning approaches suffer in the presence of view disagreement,i.e., when samples in each view do not belong to the same class due to view corruption, occlusion or other noise processes. In this paper we present a multi-view learning approach that uses a conditional entropy criterion to detect view disagreement. Once detected, samples with view disagreement are filtered and standard multi-view learning methods can be successfully applied to the remaining samples. Experimental evaluation on synthetic and audio-visual databases demonstrates that the detection and filtering of view disagreement considerably increases the performance of traditional multi-view learning approaches.
1 Introduction
Multi-view learning can exploit complementary views, but corruption may cause paired observations to belong to different classes and undermine learning. The paper proposes conditional-entropy-based detection and filtering of view disagreement, including for audio-visual learning.
1 Introduction
- Independent corruption can make a view’s class unreliable and cause paired views to disagree, challenging methods that assume each sample belongs to the same class across views.The paper models corrupted observations as a background class that can co-occur with uncorrupted observations in other views.Source: §1 · p. 1
- The proposed approach uses joint view statistics and conditional entropy to detect disagreement, then filters affected samples before multi-view learning.It assumes foreground samples co-occur with their class or background, while background samples can co-occur with any class.Source: §1 · pp. 1–2
- Audio-visual disagreement can arise from temporary occlusion or unimodal expressions, such as saying “yes” without a head nod.Source: §1 · p. 1
- Experiments on audio-visual data report that detecting and filtering disagreement enables multi-view learning despite large amounts of disagreement.Source: §1 · p. 2
2 Multi-View Learning
Conventional multi-view learning relies on agreement and largely sufficient views, assumptions that view disagreement can violate. The paper distinguishes this problem from previously studied insufficiency and previews conditional-entropy filtering as a way to retain successful learning under disagreement.
2 Multi-View Learning
- Prior multi-view work includes co-training, co-EM, co-boosting, co-regularization, and Bayesian co-training.The cited approaches use bootstrapping, soft labels, agreement objectives, consensus regularization, or a multi-view kernel.Source: §2 · p. 2
- Multi-view methods train classifiers to agree on unlabeled data, but many approaches assume each view is largely sufficient for classification.Methods addressing partial insufficiency still rely on that largely sufficient-view assumption.Source: §2 · p. 2
- View disagreement is a distinct insufficiency: paired views may belong to different classes, rather than both views being necessary to classify the sample.The paper cites examples including audio-visual speech and gesture, text and video, and webpage content and hyperlinks.Source: §2 · pp. 2–3
- Above 40% view disagreement, conventional co-training begins to diverge, while the proposed approach performs well across disagreement levels.The synthetic comparison averages performance over 10 random training-data splits, with error bars showing ±1 standard deviation.Source: Figure 2 · p. 3, §2 · p. 3
- The paper previews conditional-entropy detection as a pre-filtering step intended to let multi-view learning succeed despite substantial disagreement.Source: §2 · p. 3
3 Detection and Filtering of View Disagreement
The method uses conditional view entropy to detect disagreement, based on the assumption that background can co-occur with any class while foreground co-occurs only with the same class or background. It thresholds entropy to classify samples and applies the criterion within multi-view bootstrapping.
3 Detection and Filtering of View Disagreement
- Background may co-occur with any class, whereas foreground co-occurs only with the same class or background; conditioning on background is expected to yield higher entropy.The stated co-occurrence assumption motivates the expected entropy difference.Source: §3 · pp. 3–4
- Conditional view entropy H(x_i|x_j) measures uncertainty in one view given another and serves as an agreement cue for detecting view disagreement.The paper names H(x_i|x_j) conditional view entropy.Source: §3 · p. 3
- The method thresholds conditional view entropy at its mean over unlabeled data, using an indicator to detect foreground through low entropy.The mean is computed over unlabeled samples; foreground samples are expected to have low conditional view entropy.Source: §3 · p. 4
- A multi-view sample is in view disagreement when it is neither a redundant foreground sample nor a redundant background sample.The method distinguishes redundant foreground and background samples before identifying disagreement.Source: §3 · p. 4
- Conditional probabilities are estimated with a multivariate kernel density estimator, though the approach is agnostic to the probability model.The paper notes that more sophisticated conditional-probability models may perform better in high-dimensional input spaces.Source: §3 · p. 4
- The criterion is embedded in iterative multi-view bootstrapping, which trains and evaluates classifiers, processes high-confidence unlabeled samples, and repeats up to T iterations.The algorithm sorts unlabeled samples by classifier confidence and uses N samples in each iteration.Source: §3 · p. 5
4 Multi-view Bootstrapping in the Presence of View Disagreement
The method extends multi-view bootstrapping with conditional-entropy filtering before cross-view labeling. It can learn accurate classifiers despite severe disagreement, including in a toy example with up to 90% disagreement.
4 Multi-view Bootstrapping in the Presence of View Disagreement
- The section situates this procedure within bootstrapping, where classifiers iteratively label corresponding samples across views from confidently classified examples.The classifiers begin from a small labeled seed set.Source: §4 · p. 5
- Before cross-view labeling, the algorithm checks conditional entropy and maintains a separate labeled set for each view.It self-trains each classifier on all unlabeled examples but enforces consensus only on samples with view agreement.Source: §4 · p. 5
- The toy example shows accurate classifiers in both views despite up to 90% view disagreement.The experiment used N = 6 and continued until all unlabeled data had been used.Source: §4 · p. 5
- Cross-modality bootstrapping selects the N most-confident examples from the source classifier, then trains the target classifier on samples detected as redundant foreground or background.In audio-visual settings with unequal view difficulty, labels from the stronger classifier provide a more structured input to conditional entropy than the raw signal.Source: §4 · p. 5
5 Experimental Evaluation
Experiments evaluate audio-visual user-agreement recognition with simulated disagreement in audio and video. The proposed approach succeeds across substantial disagreement, while conventional bootstrapping or co-training degrades at higher levels.
5 Experimental Evaluation
- The evaluation uses 15 subjects performing speech-and-head-gesture responses, with simulated visual disagreement from no-motion noise or real background motion and simulated audio background babble.Cross-modality results average splits of 10 training and 5 test subjects; audio-visual co-training also varies labeled seeds and unlabeled training splits.Source: §5 · pp. 5–6, §5.1 · p. 6, §5.2 · p. 7
5.1 Cross-Modality Bootstrapping
- With real visual background motion, the approach learns a visual classifier despite significant disagreement up to 50%, unlike conventional bootstrapping.With no-motion Gaussian background, the approaches perform similarly at up to 20% disagreement, but conventional bootstrapping diverges at higher levels while the proposed algorithm remains accurate up to 50%.Source: §5.1 · pp. 6–7, Figure 4 · p. 7
5.2 Multi-View Bootstrapping
- The approach reliably learns accurate audio and visual classifiers across disagreement levels, while conventional co-training cannot improve over the supervised baseline in either modality at 70% disagreement.The initial visual classifier was much more accurate than the near-chance audio classifier.Source: §5.2 · p. 7
6 Discussion
The discussion relates view disagreement to the consensus assumptions of existing methods and describes detection and filtering as broadly applicable beyond bootstrapping. The reported realistic-data success depends on distinct foreground and background co-occurrence patterns.
6 Discussion
- Methods that define consensus between views may be vulnerable because disagreement violates their assumption that samples in each view share a class.The discussion notes this assumption for the manifold-based approach of Ando and Zhang and for co-clustering approaches that do not account for view sufficiency.Source: §6 · pp. 7–8
- The authors identify mutual-information-based dependency testing as an alternative direction for handling disagreement, rather than a method evaluated in this work.They cite hidden-factorization Markov-model hypothesis testing for audio-visual streams and leave applying such techniques to multi-view learning for future work.Source: §6 · p. 8
- The approach is demonstrated for bootstrapping, but the authors describe it as generally applicable to other methods that maximize consensus across views.They expect adaptation to other approaches to be straightforward, though those adaptations are not demonstrated here.Source: §6 · p. 8
- On the realistic dataset, success depends on foreground and background classes exhibiting distinct co-occurrence patterns that the algorithm can exploit.Figure 5 reports successful visual-classifier learning with real background and up to 50% disagreement, while conventional bootstrapping performs poorly.Source: §6 · p. 8, Figure 5 · p. 8
7 Conclusions and Future Work
The paper identifies view disagreement as a multi-view learning problem and presents conditional entropy for detecting disagreement caused by corruption or noise. Audio-visual user-agreement experiments succeed at 50–70% disagreement; future work includes alternative thresholding and probability models.
7 Conclusions and Future Work
- For audio-visual user agreement, the method performs multi-view learning with 50–70% view disagreement, and future work includes alternative entropy thresholds, probability models, and mutual information for nonstationary redundancy.The conclusion describes the framework as semi-supervised and the entropy criterion as detecting disagreement caused by view corruption or noise.Source: §7 · p. 8