PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-view Vector-valued Manifold Regularization for Multi-label Image Classification

Yong Luo, Dacheng Tao, Chang Xu, Chao Xu, Hong Liu, Yonggang Wen

arXiv:1904.03921v1stat.MLcs.CVcs.LG

TL;DR

Multi-label image classification requires modeling label relationships while integrating complementary visual views, but existing methods often neglect one of these aspects. The paper proposes MV3MR, with an SVM implementation that learns view weights and uses shared multi-view geometry. On VOC and MIR, MV3LSVM generally outperforms multi-view and multi-label alternatives, consistently in mAP and with the best average ranks across three criteria.

  • Problem

    Multi-label images require both label-dependency modeling and multiple feature views, whereas existing approaches often omit feature combination or assume equal view contributions.

  • Method

    MV3MR integrates view-specific kernels into a vector-valued manifold-regularization model, learns their weights, and yields the MV3LSVM SVM formulation.

  • Results

    MV3LSVM generally outperforms multi-view and multi-label methods, consistently achieves higher mAP, and has the smallest average rank for mAP, mAUC, and ranking loss.

  • Takeaways & Limitations

    Combining complementary views with label correlations and shared local geometry is effective for multi-label image classification on VOC and MIR.

Abstract

from arXiv · show

In computer vision, image datasets used for classification are naturally associated with multiple labels and comprised of multiple views, because each image may contain several objects (e.g. pedestrian, bicycle and tree) and is properly characterized by multiple visual features (e.g. color, texture and shape). Currently available tools ignore either the label relationship or the view complementary. Motivated by the success of the vector-valued function that constructs matrix-valued kernels to explore the multi-label structure in the output space, we introduce multi-view vector-valued manifold regularization (MV$\mathbf{^3}$MR) to integrate multiple features. MV$\mathbf{^3}$MR exploits the complementary property of different features and discovers the intrinsic local geometry of the compact support shared by different features under the theme of manifold regularization. We conducted extensive experiments on two challenging, but popular datasets, PASCAL VOC' 07 (VOC) and MIR Flickr (MIR), and validated the effectiveness of the proposed MV$\mathbf{^3}$MR for image classification.

I. INTRODUCTION

Multi-label image classification must model related labels and combine complementary visual features, but existing approaches often address only one of these structures. The paper introduces MV3MR and its SVM formulation, MV3LSVM, then evaluates them on VOC and MIR.

  • I. INTRODUCTION: Related labels such as person–motorbike, sea–sky, and clouds–sky make image classification intrinsically different from independent binary prediction.
  • I. INTRODUCTION: Different features capture different image properties, so no single representation effectively describes all natural objects, man-made objects, and scenes.
  • I. INTRODUCTION: Existing vector-valued classification models capture label dependencies but cannot directly handle multi-view images, while feature concatenation can ignore feature meaning and overfit with few samples.
  • I. INTRODUCTION: MV3MR integrates multiple kernels into the vector-valued function, weighting more discriminative views more heavily while exploiting their complementary information.
  • I. INTRODUCTION: MV3MR assembles unlabeled multi-view data to estimate shared local geometry, and MV3LSVM learns kernel weights within an SVM formulation.
  • I. INTRODUCTION: Experiments on PASCAL VOC’07 and MIR Flickr compare MV3LSVM with multi-kernel and multi-label baselines using mAP, mAUC, and hamming loss.

B. MKL: Multi-kernel learning

The paper motivates multi-kernel learning for multi-label image classification because different views need not contribute equally. It combines manifold regularization, vector-valued kernels, and learned kernel weights to represent label structure and multi-view geometry.

  • C. Image classification: Most existing approaches do not jointly model multi-view representation and label interdependencies, and many multi-view methods assume equal feature contributions.
  • IV. MV3MR: MULTI-VIEW VECTOR-VALUED MANIFOLD REGULARIZATION: MV3MR instead learns combination coefficients for views, exploiting complementary features while preserving label correlations under manifold regularization.
  • A. Manifold regularization: Manifold regularization uses a nearest-neighbor graph and graph Laplacian to constrain predictions according to the local geometry of the data distribution.
  • B. Vector-valued manifold regularization: The vector-valued extension represents multiple labels through a matrix-valued kernel whose output structure is derived from the label graph Laplacian.
  • B. Vector-valued manifold regularization: The illustrative kernel example indicates that combining views can produce a Gram matrix closer to the optimal matrix than simply averaging kernels.

IV. MV3MR: MULTI-VIEW VECTOR-VALUED MANIFOLD REGULARIZATION

MV3MR generalizes vector-valued manifold regularization to multi-view, multi-label image classification by learning complementary view contributions while preserving label correlations and manifold locality.

  • IV. MV3MR: MULTI-VIEW VECTOR-VALUED MANIFOLD REGULARIZATION: The framework extends VVMR by allowing different views to contribute unequally instead of assuming equal contributions.
  • IV. MV3MR: MULTI-VIEW VECTOR-VALUED MANIFOLD REGULARIZATION: An illustrative two-view example shows the learned combined Gram matrix is closer to the optimal Gram matrix than the average Gram matrix.
  • IV. MV3MR: MULTI-VIEW VECTOR-VALUED MANIFOLD REGULARIZATION: Label information constructs an output similarity graph whose Laplacian is incorporated into each view’s Gram matrix to enforce label correlations.
  • IV. MV3MR: MULTI-VIEW VECTOR-VALUED MANIFOLD REGULARIZATION: MV3MR learns view-specific kernel and graph weights, then combines their Gram matrices and Laplacians for classification on an integrated manifold.Each view contributes a Gram matrix Gv and graph Laplacian Mv; the learned weights βv and θv form G and M.

A. Rationality

The multi-view construction places scalar kernels and vector-valued label structure in a shared RKHS, while jointly learning view weights and classifier coefficients under manifold regularization.

  • A. Rationality: Each view defines a positive-definite scalar kernel, and nonnegative coefficients combine the resulting RKHSs into a multi-view kernel space.
  • A. Rationality: The vector-valued kernel combines view-specific kernels with the output-space matrix Q, thereby incorporating label relationships into the multi-view function space.
  • A. Rationality: MV3MR learns a vector-valued function by linearly combining kernels and graph Laplacians from different views under manifold regularization.
  • A. Rationality: The integrated graph Laplacian is a weighted combination of the view-specific vector-valued Laplacians Mv.
  • A. Rationality: The method adopts hinge loss because it is described as more suitable for classification than least-squares loss.

C. Multi-view vector-valued Laplacian SVM

MV3LSVM solves MV3MR through an SVM formulation that alternates classifier, kernel-weight, and graph-weight updates, with the weights constrained to nonnegative simplices.

  • C. Multi-view vector-valued Laplacian SVM: The optimization jointly estimates classifier coefficients, kernel weights β, and graph-Laplacian weights θ using combined Gram and manifold matrices.
  • C. Multi-view vector-valued Laplacian SVM: Coordinate descent updates β under nonnegative simplex constraints, favoring views with greater discriminative ability and lower manifold-regularization cost.
  • C. Multi-view vector-valued Laplacian SVM: The β-update has a closed-form stationary solution but uses coordinate descent because direct projection lacks convergence guarantees and may cause numerical problems.
  • C. Multi-view vector-valued Laplacian SVM: The learned β weights are not strictly consistent with the performance of each view trained separately because the joint solutions need not coincide.
  • C. Multi-view vector-valued Laplacian SVM: The algorithm alternates between solving for classifier coefficients, updating kernel weights β, and updating graph weights θ.

D. Convergence analysis

The convergence analysis establishes strict convexity for the three block subproblems and uses successive objective decreases to guarantee convergence of the alternating algorithm.

  • D. Convergence analysis: Strict convexity is established for the classifier subproblem using positive-definite G and positive-semidefinite manifold regularization.
  • D. Convergence analysis: The kernel-weight subproblem is strictly convex because its Hessian combines the symmetric matrix H with γB I.
  • D. Convergence analysis: The graph-weight subproblem is strictly convex when γC > 0.
  • D. Convergence analysis: The alternating MV3LSVM updates produce nonincreasing objective values across classifier, kernel-weight, and graph-weight steps, guaranteeing convergence.

E. Complexity analysis

MV3LSVM’s complexity is dominated by repeatedly computing the classifier variable, while view-weight updates remain efficient because their dimensionality is small.

  • Classifier computation dominates each MV3LSVM iteration, requiring O(n^2.8N^2.8) for matrix operations and O(n^2.3l^2.3) with an SMO-based SVM solver.The matrix S computation involves inversion and multiplications of an nN × nN matrix.
  • The β and θ updates are relatively efficient because their dimensionality is V, which was only 7 in the experiments.

V. EXPERIMENT

Experiments evaluate MV3LSVM on VOC and MIR using multiple image representations, validation-based tuning, and varying numbers of labeled training samples.

  • VOC contains 10,000 images across 20 categories, while MIR contains 25,000 images covering 38 concepts.VOC uses a standard 5,011-image training and 4,952-image test split; MIR is split into equally sized training and test sets.
  • The experiments use 20% of each test set for validation, with parameters selected on validation performance before transductive and inductive evaluation.
  • Ten random choices of l ∈ {100, 200, 500} labeled training samples are evaluated.
  • Seven representations combine SIFT, hue, GIST, RGB, HSV, LAB, and tag features into separate views.The visual representations include local descriptors, a global descriptor, color histograms, and tag features.
  • The views use distance-specific kernels: L1 for color histograms, L2 for GIST, χ2 for SIFT and hue histograms, and a linear kernel for tags.

A. Evaluation metrics

Evaluation measures ranking quality for individual labels and label-set prediction using AP, AUC, and ranking loss, with mean scores reported across labels or samples.

  • Average precision evaluates how highly positive samples are ranked for each label, and the reported mAP averages AP across labels.The paper uses the PASCAL VOC evaluation procedure, where larger values indicate better performance.
  • AUC measures the probability that a classifier ranks a positive sample above a negative one using the ROC curve.A realistic classifier should have AUC greater than 0.5.
  • Ranking loss evaluates label-set prediction by counting incorrectly ordered positive-negative label pairs for each instance.Smaller ranking-loss values indicate better performance, and the mean is computed across samples.
  • Objective values are tracked across iterations in Fig. 5 for VOC and MIR.

B. Performance enhancement with multi-view learning

Learning view-combination weights improves performance over uniform weighting, while the method converges in few iterations, remains initialization-insensitive, and generalizes from unlabeled to unseen data.

  • Learning combination weights is always superior to uniform view weights on both datasets, although the improvement shrinks as labeled samples increase.The experiments compare mAP, mAUC, and RL for learning β alone versus learning both β and θ.
  • β has more influence on final performance overall than θ, according to the reported observations.
  • MV3LSVM requires about five iterations to obtain a satisfactory solution, making its time complexity only slightly higher than VVLSVM.
  • Performance varies little across 10 random initializations of β and θ on both datasets.The stability is assessed using mAP, mAUC, and RL.

C. Out-of-sample generalization

MV3LSVM generalizes from labeled and unlabeled data to unseen images while outperforming competing multi-view and multi-label methods on VOC and MIR. Its learned view and graph weights generally track individual-view mAP while exploiting complementary information across views.

  • C. Out-of-sample generalization: The method generalizes well from the unlabeled training set to unseen images, retaining strong out-of-sample performance across VOC and MIR.The comparison uses per-label AP scores across 10 random labeled-data choices with l = 200.
  • D. Analysis of the combination coefficients in multi-view learning: View and graph weights generally increase with each view’s mAP, although the integrated model can assign greater weight to a less discriminative view.DenseSIFT and tags illustrate the usual alignment, while HSV receives more weight than DenseSIFT because the coefficients are learned jointly.
  • E. Comparisons with multi-label and multi-kernel learning algorithms: The final evaluation compares MV3LSVM with feature concatenation, uniform-kernel SVM, SimpleMKL, and other multi-label algorithms on VOC and MIR.The criteria are mAP, mAUC, and hamming loss.
  • E. Comparisons with multi-label and multi-kernel learning algorithms: MV3LSVM consistently outperforms competing multi-view and multi-label methods in mAP, with the smallest average rank across mAP, mAUC, and HL-related criteria.On VOC with 100 labeled samples, it improves mAP over SimpleMKL by 8.1%.

APPENDIX A PROOF OF LEMMA 1

The lemma establishes that the integrated matrix M is a vector-valued graph Laplacian built from view-specific adjacency graphs. Under manifold regularization, functions can be represented in the span of kernels centered on the sampled manifold points.

  • APPENDIX A PROOF OF LEMMA 1: M is a convex combination of scalar graph Laplacians from different views, making it the corresponding vector-valued graph Laplacian.Each view graph uses k-nearest neighbors and view-specific similarities.
  • APPENDIX A PROOF OF LEMMA 1: The intrinsic norm induced by M depends only on a function’s restriction to the manifold, so components vanishing there do not affect the regularizer.The proof decomposes f into components in the kernel span and its orthogonal complement, then shows the latter vanishes on the manifold.
  • APPENDIX A PROOF OF LEMMA 1: For fixed view weights β and θ, the optimization minimizer lies in the span of kernels centered at the labeled and unlabeled samples.This is the representer-style conclusion used to express the solution through finite sample coefficients.
Loading 1904.03921v1…