Source-linked AI summary
Majority Vote of Diverse Classifiers for Late Fusion
Emilie Morvant, Amaury Habrard, Stéphane Ayache
TL;DR
Late fusion combines modality scores at decision level, and selecting useful weights involves accounting for dependencies between views and voter diversity. The paper adapts MinCq with an order-preserving pairwise loss to optimize MAP while accounting for diversity. On PASCAL VOC’07, linear MinCq methods outperform linear baselines on average, while kernel MinCq methods achieve the highest MAP for every concept against SVM.
Problem
Late fusion combines modality scores at decision level, and selecting useful weights involves accounting for dependencies between views and voter diversity.
Source: §1 · pp. 1–2, §3.1 · p. 4Method
The paper adapts MinCq for late fusion with an order-preserving pairwise loss that optimizes ranking preferences while taking voter diversity into account.
Source: §1 · p. 2, §3.2 · p. 5Results
On PASCAL VOC’07, linear MinCq methods outperform linear baselines on average, while kernel MinCq methods achieve the highest MAP for every concept versus SVM.
Source: §4 · pp. 8–9Takeaways & Limitations
The experiments support MinCq-based methods as a suitable approach for late fusion of predictions from classifiers trained on different modalities.
Source: §5 · p. 9The full pairwise version introduces m+×m− additive variables, making the optimization harder to solve and motivating a relaxation.
Source: §3.2 · p. 6
Abstract
from arXiv · showhide
In the past few years, a lot of attention has been devoted to multimedia indexing by fusing multimodal informations. Two kinds of fusion schemes are generally considered: The early fusion and the late fusion. We focus on late classifier fusion, where one combines the scores of each modality at the decision level. To tackle this problem, we investigate a recent and elegant well-founded quadratic program named MinCq coming from the machine learning PAC-Bayesian theory. MinCq looks for the weighted combination, over a set of real-valued functions seen as voters, leading to the lowest misclassification rate, while maximizing the voters' diversity. We propose an extension of MinCq tailored to multimedia indexing. Our method is based on an order-preserving pairwise loss adapted to ranking that allows us to improve Mean Averaged Precision measure while taking into account the diversity of the voters that we want to fuse. We provide evidence that this method is naturally adapted to late fusion procedures and confirm the good behavior of our approach on the challenging PASCAL VOC'07 benchmark.
1 Introduction
The introduction motivates late fusion as combining modality scores at decision level and proposes MinCq-based fusion that accounts for voter diversity and ranking preferences. The proposed ranking extension uses pairwise constraints to target Mean Average Precision.
1 Introduction
- Late fusion combines modality prediction scores at decision level, whereas early fusion merges features before learning and classification.The introduction notes that heterogeneous features can be hard to combine in early fusion.Source: §1 · p. 1
- The paper frames late fusion as a weighted majority vote over classifier scores, with weights chosen to account for dependencies between views and voter diversity.The introduction notes that AdaBoost can degrade fusion performance when combining strong classifiers.Source: §1 · p. 2
- MinCq learns a weighted majority vote over real-valued voters by balancing error risk and voter diversity, and the paper applies it to classifier fusion.The framework is presented as a quadratic program with theoretical guarantees.Source: §1 · p. 2
- For multimedia retrieval, the authors extend MinCq with an order-preserving pairwise loss to optimize Mean Average Precision while accounting for voter diversity.The loss checks pairwise ranking constraints.Source: §1 · p. 2
2 MinCq: A Quadratic Program for Majority Votes
MinCq is a quadratic program for learning a weighted majority vote over real-valued voters in binary classification. It minimizes an empirical C-bound objective under weight constraints, using the Q-margin to characterize classification confidence.
2 MinCq: A Quadratic Program for Majority Votes
- MinCq considers binary classification with examples drawn i.i.d. from an unknown distribution and learns weights over real-valued voters to form a weighted majority vote.Each voter predicts by the sign of its real-valued output.Source: §2 · p. 3
- The Q-margin, yH_Q(x), models confidence in an example's label; a strictly positive Q-margin corresponds to a correct classification.The majority vote's risk is its probability of misclassifying an example drawn from the data distribution.Source: §2 · p. 3
- MinCq minimizes the empirical counterpart of the C-bound, a bound on majority-vote risk justified by a PAC-Bayesian generalization bound.The C-bound is established using the Cantelli-Chebitchev inequality.Source: §2 · pp. 2–3
- With the Q-margin fixed, MinCq minimizes the C-bound denominator under a regularization constraint, yielding a quadratic program.The weights satisfy an L1-norm constraint and are individually bounded between -1 and 1.Source: §2 · pp. 3–4
3 A New Framework for Classifier Late Fusion
The framework extends MinCq’s diversity-aware weighted voting to late classifier fusion and adapts it to ranking. It proposes pairwise-preference and average-negative approaches for improving MAP while balancing voter diversity and ranking relevance.
3.1 Justification of MinCq as a Classifier Late Fusion Algorithm
- MinCq favors maximally uncorrelated voters, making its diversity objective a rationale for fusing classifiers trained on different modalities.The method’s second-moment objective is described as maximizing diversity between voters.Source: §3.1 · pp. 4–5
3.2 MinCq for Ranking
- The pairwise-preference approach uses an order-preserving hinge loss to rank positive examples above negative ones.Its quadratic program adds m+ × m− slack variables weighted by β > 0.Source: §3.2 · pp. 5–6
- The pairwise formulation has a computational drawback: its m+ × m− additive variables make the problem harder to solve.The paper introduces a relaxation to address this drawback.Source: §3.2 · p. 6
- The average-preference alternative constrains positive examples to score above the average negative score, using only m+ additional variables.It relaxes the pairwise constraints by considering the average negative score.Source: §3.2 · p. 6
- MinCq’s ranking extension favors weighted votes that balance voter diversity with relevant rankings.Both ranking approaches retain the original MinCq framework.Source: §3.2 · p. 6
4 Experiments on PascalVOC’07 benchmark
Experiments on PascalVOC’07 compare MinCq-based late-fusion methods with baselines. With an RBF kernel layer, MinCq methods achieved the highest MAP for every concept, and the averaged pairwise-preference variant was significantly best.
4 Experiments on PascalVOC’07 benchmark
- The benchmark contains 10,000 images and 20 concepts; training used all positive examples plus sampled negatives at a 1:2 ratio, while testing used the original test set.The experiments used nine visual features, with an SVM score function trained for each feature.Source: §4 · pp. 6–7
- With an RBF kernel layer, MinCq-based methods achieved the highest MAP for every concept against SVM, and MinCqrbf PW av was significantly best (p=0.0003 vs SVM; p=0.0038 vs MinCqrbf).The authors describe the averaged pairwise-preference loss as a compromise between improving MAP and computational cost.Source: Table 2 · p. 8, §4 · p. 9
- A second experiment added an RBF kernel layer over examples represented by their nine SVM scores, using sample kernels as voters and comparing against classical SVM stacking.Hyperparameters were selected by five-fold cross-validation to maximize MAP.Source: §4 · p. 8
- In the linear experiments, MinCq-based methods had the highest MAP for 16 of 20 concepts and significantly outperformed the linear baselines on average.The reported p-values versus Σ were 0.0139, 0.0232, and 0.0088 for three MinCq variants; against ΣMAP, best, and hbest, p<0.001.Source: Table 1 · p. 7, §4 · p. 8
- The average-negative relaxation performed similarly to full pairwise preferences (p=0.223), while classical MinCq had the best MAP among the first-series methods (p=0.2574 for the hinge-loss comparison).The authors report that the order-preserving hinge loss was not helpful in that setting.Source: §4 · p. 8
5 Conclusion and Perspectives
The paper adapts MinCq’s diversity-aware weighted majority vote for multimedia late fusion with ranking preferences. Its experiments support MinCq as a competitive alternative for image indexing, while the conclusion outlines theoretical and empirical directions for further work.
5 Conclusion and Perspectives
- The authors adapt MinCq, originally developed for binary classification, to ranking by incorporating pairwise preferences while accounting for model diversity.They conclude that the extension is appropriate for combining predictions from classifiers trained on different modalities in late fusion.Source: §5 · p. 9
- The experiments support MinCq as a competitive alternative for classifier fusion in image indexing.The authors propose investigating a ranking-specific C-bound and other diversity metrics to study the trade-off between ranking quality and input diversity.Source: §5 · p. 9
- A further proposed direction is to use PAC-Bayesian prior beliefs over classifiers when learning majority votes for ranking.The paper notes that PAC-Bayesian guarantees relate the vote to the prior through Kullback-Leibler divergence.Source: §5 · pp. 9–10