Source-linked AI summary
Multi-Label Learning with Global and Local Label Correlation
Yue Zhu, James T. Kwok, Zhi-Hua Zhou
TL;DR
Existing methods typically model either globally shared or locally shared label correlations, and partial labels make correlation estimation harder because the observed distribution differs from the true one. GLOCAL jointly recovers missing labels, trains classifiers, and learns both correlation types through latent label representations and label manifolds. Experiments report that it outperforms state-of-the-art approaches on both full-label and missing-label learning.
Problem
Existing approaches separately assume global or local label correlations, while missing labels make correlation estimation difficult because the observed label distribution differs from the true one.
Method
GLOCAL jointly recovers missing labels, trains classifiers, and exploits global and local correlations by learning latent label representations and optimizing label manifolds.
Results
GLOCAL outperforms state-of-the-art multi-label learning approaches on both full-label and missing-label data.
Takeaways & Limitations
The approach provides a unified multi-label learning framework for simultaneously handling full-label and missing-label cases with global and local correlations.
Takeaways & Limitations
The work handles symmetric label correlations and identifies asymmetric correlations as future work.
Abstract
from arXiv · showhide
It is well-known that exploiting label correlations is important to multi-label learning. Existing approaches either assume that the label correlations are global and shared by all instances; or that the label correlations are local and shared only by a data subset. In fact, in the real-world applications, both cases may occur that some label correlations are globally applicable and some are shared only in a local group of instances. Moreover, it is also a usual case that only partial labels are observed, which makes the exploitation of the label correlations much more difficult. That is, it is hard to estimate the label correlations when many labels are absent. In this paper, we propose a new multi-label approach GLOCAL dealing with both the full-label and the missing-label cases, exploiting global and local label correlations simultaneously, through learning a latent label representation and optimizing label manifolds. The extensive experimental studies validate the effectiveness of our approach on both full-label and missing-label data.
1. Introduction
Multi-label data can exhibit both globally shared and locally specific label correlations, while missing labels make those correlations difficult to estimate. GLOCAL jointly recovers missing labels, trains classifiers, and learns both correlation types through latent label representations and label manifolds.
- Multi-label instances may carry multiple labels, and label correlations can be global across all instances or local to a data subset.
- Label correlations vary across contexts, lack a unified specification measure, and may be difficult to estimate manually.
- Hierarchical label structures may not exist in applications such as scene annotation, while co-occurrence estimates can overfit and become unreliable for rare labels.
- Missing labels alter the observed label distribution, making correlation estimates difficult and biasing methods based on hierarchy or co-occurrence.
- GLOCAL simultaneously recovers missing labels, trains linear classifiers, and exploits global and local label correlations.
- The method learns correlations without external knowledge by jointly learning latent label representations and the instance-label mapping, encouraging classifier outputs to reflect positive or negative relations.
2. Related Work
Related work includes first-, second-, and high-order correlation strategies, local-correlation methods, and low-rank approaches for missing labels. These methods leave open the simultaneous learning of global and local correlations, especially with partially observed labels.
- Correlation strategies: Multi-label methods use first-order independent classifiers, second-order pairwise relations, or high-order models incorporating other labels’ influences.
- Local correlations: MLLOC models local correlations by embedding label influence into each instance’s feature representation.
- Local correlations: MLLOC becomes less discriminative in high-dimensional feature spaces, omits global correlations, and cannot learn with missing labels.
- Missing labels: Low-rank methods address missing labels through matrix completion, latent structure, or supplementary label matrices, but have setting or correlation-scope limitations.
- Research gap: Existing approaches generally do not learn global and local label correlations simultaneously, motivating a unified approach.
- Label manifolds: Label-manifold regularization transfers instance-similarity ideas to labels, but its Laplacian depends on a label similarity or correlation matrix that can be hard to specify.
3. The Proposed Approach
GLOCAL learns a latent label representation while jointly exploiting global and local label correlations, including when training labels are only partially observed.
- Problem setting: GLOCAL addresses multi-label learning with partially observed labels, where both positive and negative labels may be missing.
- Latent label representation: The method factorizes the low-rank ground-truth label matrix into latent labels and a projection from original labels to the latent space.V contains compact, semantically abstract latent labels, while U projects original labels into that space.
- Basic model: GLOCAL learns classifier outputs from instances through a linear mapping and reconstructs observed labels using only the observed entries.The observation indicator J selects labeled entries in the reconstruction objective.
- Global and local manifold regularization: Global and local label manifolds regularize classifier outputs so positively correlated labels are similar and negatively correlated labels are dissimilar.Local groups may be specified by domain knowledge or obtained through clustering.
- Optimization: The optimization jointly combines label reconstruction, latent representation fitting, regularization, and global and local manifold terms.An alternating algorithm updates Z, V, U, and W until convergence or a maximum iteration count.
- Learning label manifolds: Because observed label correlations can be noisy or misleading with few positives or missing labels, GLOCAL learns Laplacian matrices directly instead of specifying correlation metrics.The learned matrices are decomposed as ZmZm^T and constrained to avoid the trivial zero solution.
3.3. Learning by Alternating Minimization
GLOCAL solves its objective by alternating updates of manifold, latent-label, projection, and classifier variables, using gradient-based optimization and closed-form subproblems where available.
- Alternating minimization: Alternating minimization updates one variable among Z, U, V, and W while keeping the others fixed.Updates use gradient descent, with line search implemented through MANOPT for the relevant variables.
- Updating Z: The Zm update uses projected gradient descent under unit-norm row constraints imposed by diag(ZmZm^T) = 1.Each row of Zm is projected onto the unit norm ball.
- Updating V: Each column of V can be optimized independently, yielding a closed-form solution that may be replaced by gradient descent when matrix inverses are expensive.
4. Experiments
The experiments evaluate GLOCAL on text and image datasets in both full-label and missing-label settings.
- Experiments use text and image datasets to assess performance in both full-label and missing-label cases.
4.1. Setup
The setup evaluates GLOCAL against several state-of-the-art multi-label methods on Yahoo, Enron, Corel5k, and Image3 datasets using repeated train-test splits and ranking metrics.
- Datasets: The benchmark contains eleven Yahoo text datasets, Enron, Corel5k, and Image3 image data.Dataset information is reported in Table 1.
- Data split: Each dataset is split by randomly assigning 60% of instances to training and the remainder to testing.
- Compared methods: GLOCAL partitions data into local groups using k-means clustering, while the comparison includes BR, MLLOC, LEML, and ML-LRC.The baselines represent no label-correlation modeling, local correlations, implicit global correlations, and low-rank global correlations, respectively.
- Evaluation protocol: Results are averaged over 10 independent repetitions, with parameters selected by 5-fold cross-validation on the training set.
- Metrics: Evaluation uses ranking loss and coverage, where lower is better, plus average AUC and average precision, where higher is better.
4.2. Learning with Full Labels
On full-label data, GLOCAL models global and local label correlations together, improving performance overall and on small local clusters. The Image example shows that local correlations vary across groups and differ from the learned global correlation.
- BR performs worst because it treats each label independently.
- GLOCAL is the best overall because it models both global and local label correlations.
- Local label correlations vary across Image groups and differ from the global correlation.
- In group 1, sunset is highly correlated with desert, while group 2 more often combines mountain and sea.
- The learned global correlation positively links sea with sunset, mountain with trees, and negatively links desert with sea and trees.
- On small clusters, using both correlation types achieves comparable or even better performance, whereas either global or local correlation alone can suffice on some datasets.
4.3. Learning with Missing Labels
GLOCAL addresses partially observed labels by jointly recovering missing labels, learning classifiers, and using global and local label correlations. Across different observation rates, it performs best overall.
- GLOCAL performs best at different observed-label rates because it simultaneously uses global and local correlations with label manifold regularization.
- Performance generally increases as more label-matrix entries are observed.
- GLOCAL jointly handles label recovery and classifier learning, unlike methods that separate these stages.
- Methods using only global correlations are often worse than GLOCAL, while BR and MLLOC require full labels without preprocessing.
4.4. Convergence
GLOCAL’s objective converges quickly on the evaluated datasets. For missing-label learning, GLOCAL is among the fastest methods, although its clustering step makes it slower than LEML.
- The objective converges quickly in a few iterations on Arts, Business, Enron, and Image, with similar behavior on other datasets.
- GLOCAL and LEML are fastest for missing-label learning because they jointly train classifiers and exploit low-rank structure.
- GLOCAL is slower than LEML because it requires an additional clustering step to obtain local groups.
- GLOCAL outperforms LEML in label recovery, while ML-LRC is slower because it does not reduce the label matrix or model size during iterations.
- MBR and MMLLOC can be slowest because they train a classifier for each label and may additionally recover missing labels.
4.5. Sensitivity to Parameters
GLOCAL is sensitive to the balance between local and global modeling and to latent dimensionality. Too many clusters or an oversized latent representation can reduce performance, while λ2 has little effect.
- The sensitivity study varies g, λ3, λ4, λ2, and latent dimensionality k while holding other parameters at their best settings.
- Performance improves as more clusters incorporate local correlations, then deteriorates when clusters become too small for reliable estimation.
- Setting λ3 = 0 leaves only local correlations and produces poor performance; analogous behavior occurs when λ4 eliminates local-correlation emphasis.
- Increasing latent dimensionality k initially improves performance, but excessive k worsens it because the low-rank structure is not fully utilized.
- GLOCAL is not sensitive to the Frobenius-norm regularization parameter λ2.
5. Conclusion
GLOCAL jointly exploits global and local label correlations while recovering missing labels and training a classifier through latent label representations and optimized label manifolds. It provides a unified solution for full-label and missing-label learning and outperforms state-of-the-art approaches in both settings, while currently assuming symmetric label correlations.
- GLOCAL simultaneously recovers missing labels, trains the classifier, and exploits global and local label correlations.Its approach learns a latent label representation and optimizes label manifolds.
- GLOCAL is the first approach described here to exploit both global and local label correlations without requiring prior knowledge of those correlations.It directly learns the Laplacian matrix so classifier outputs and label correlations match globally and locally.
- GLOCAL provides a unified solution for both full-label and missing-label multi-label learning.
- Experimental results show that GLOCAL outperforms state-of-the-art multi-label learning approaches on both full-label and missing-label data.
- The current work handles symmetric label correlations, while asymmetric correlations are identified as future work.The paper gives mountain–tree relations as an example where correlation strength may differ by direction.