Source-linked AI summary
Less is more: zero-shot learning from online textual documents with noise suppression
Ruizhi Qiao, Lingqiao Liu, Chunhua Shen, Anton van den Hengel
TL;DR
Text-based zero-shot learning seeks to replace labor-intensive attributes with accessible online documents, whose textual representations are often noisy. The paper uses an l2,1-norm objective to suppress that noise while matching text and visual features, and reports superior performance over competing online-information methods on two large datasets.
Problem
Online textual documents can reduce the human labor required to create zero-shot semantic representations, but their textual representations are usually too noisy for zero-shot learning.
Method
The paper jointly suppresses noisy textual signals and learns a text–visual matching function using an l2,1-norm objective with an efficient optimization algorithm.
Results
The proposed noise-suppression method significantly outperforms competing zero-shot learning methods that rely on online information sources without explicit noise suppression.
Takeaways & Limitations
Noise in online text affects text-based zero-shot learning, while suppressing its impact can improve performance and reveal useful document information.
Abstract
from arXiv · showhide
Classifying a visual concept merely from its associated online textual source, such as a Wikipedia article, is an attractive research topic in zero-shot learning because it alleviates the burden of manually collecting semantic attributes. Several recent works have pursued this approach by exploring various ways of connecting the visual and text domains. This paper revisits this idea by stepping further to consider one important factor: the textual representation is usually too noisy for the zero-shot learning application. This consideration motivates us to design a simple-but-effective zero-shot learning method capable of suppressing noise in the text. More specifically, we propose an $l_{2,1}$-norm based objective function which can simultaneously suppress the noisy signal in the text and learn a function to match the text document and visual features. We also develop an optimization algorithm to efficiently solve the resulting problem. By conducting experiments on two large datasets, we demonstrate that the proposed method significantly outperforms the competing methods which rely on online information sources but without explicit noise suppression. We further make an in-depth analysis of the proposed method and provide insight as to what kind of information in documents is useful for zero-shot learning.
1. Introduction
Text-based zero-shot learning uses online documents to avoid manually creating attributes, but those documents introduce noise. The paper proposes an l2,1-norm method that suppresses noisy textual signals while matching text and visual features.
- Zero-shot learning recognizes objects from classes unseen during training, reducing the need to collect training data for every possible class.
- Human-specified attributes are effective semantic representations, but creating them requires substantial labor that conflicts with fully automatic recognition.
- Online textual documents such as Wikipedia offer accessible, information-rich semantic representations, but their textual content is noisy for zero-shot learning.
- Noise suppression reduces the impact of less relevant document dimensions rather than necessarily discarding them.
- The proposed l2,1-norm objective simultaneously suppresses noisy signals in text and learns a function matching textual documents with visual features.
- The method includes an efficient optimization algorithm and outperforms competing online-information methods on two large-scale zero-shot learning benchmarks.
2. Related work
Related work has expanded zero-shot learning beyond human-specified attributes using statistical, distributed, and online textual representations. Prior document-based methods directly connect text with visual features, but their reported performance has remained disappointing relative to state-of-the-art approaches.
- Earlier zero-shot learning methods commonly rely on human-specified attributes and improve how attribute–category connections are learned.
- Researchers have explored co-occurrence statistics and distributed word representations as more easily obtained alternatives or complements to manually specified attributes.
- Online textual documents such as Wikipedia articles provide another source for constructing semantic representations, including noisy web-based attribute candidates.
- Recent methods directly learn compatibility between documents and visual features, but their performance has been disappointing compared with state-of-the-art zero-shot learning methods.
3. Our Approach
The approach represents documents as binarized word histograms and learns classifiers by matching visual features with text while suppressing noisy textual dimensions. An alternating optimization solves the resulting formulation.
- Text representation: Documents are represented with a binarized bag-of-words histogram after tokenization, stop-word and punctuation removal.
- Learning to match text and visual features: The method learns a classifier-generating function that can transfer from seen classes to unseen classes through their document representations.Classifiers are generated from semantic representations, allowing unseen-class classifiers to be obtained for test-image prediction.
- Noise suppression: The document mapping Wz suppresses noisy components of the document representation, while Wx generates image classifiers from the noise-suppressed intermediate representation.The two matrices receive different regularization terms because they perform different roles.
- Noise suppression: The l2,1-norm encourages columns of Wz to contain few large values, reducing the impact of noisy document dimensions without necessarily discarding them.With sufficiently large λ2, the effect can approach feature selection, but cross-validated regularization generally retains most document dimensions.
- Optimization: The objective is nonconvex jointly but convex in Wx and Wz separately, so the algorithm alternates between solving for Wz and Wx.The Wz subproblem uses an iterative l2,1-regularized regression procedure, while the Wx subproblem is conventional least squares.
4. Experiments
Experiments on AwA and CUB-200-2011 evaluate the method against online-text, attribute-based, and state-of-the-art zero-shot learning approaches, then analyze its noise suppression and document dimensions.
- Performance evaluation: The method is compared with approaches using online textual sources, human-specified semantic attributes, and state-of-the-art zero-shot learning systems.Tables 1–3 cover Wikipedia-based comparisons, automatically mined information, and methods using low-level or deep CNN features.
- Experimental setup: Experiments use AwA and CUB-200-2011, with Wikipedia-derived text representations, two image-feature types, cross-validation, and results averaged over 10 trials.AwA has 30,475 images across 50 mammal classes; CUB-200-2011 has 11,788 images across 200 bird categories.
- Performance evaluation: 5% inferior to our approach is the reported ESZSL gap in one comparison, while the proposed method also improves by more than 4% over ESZSL in another.The paper attributes these gains to the introduced noise suppression mechanism.
- Performance evaluation: On AwA and CUB-200-2011, the method outperforms approaches using online text sources, while methods surpassing it use cleaner human-defined attributes.On CUB-200-2011, subtle bird-category differences may not be well captured by Wikipedia articles; higher-quality text is suggested as a possible improvement.
- Noise suppression analysis: The l2,1-norm assigns large importance weights to relatively few document dimensions, whereas the Frobenius norm spreads large weights across more dimensions.Removing low-importance dimensions manually produces poorer performance, indicating suppression is not equivalent to feature selection.
- Document representation analysis: Top-weighted words are often semantically meaningful, including body parts, habitat, behaviour, affordance, taxonomy, and environment, and some imply visual properties indirectly.The analysis gives “ruminant” as an example of an abstract concept connecting antelope and deer through deer-like or cattle-like visual associations.
5. Conclusion
The paper introduces noise suppression for text-based zero-shot learning and finds that textual noise affects performance, while useful information includes meaningful, weakly related, and distributional word patterns.
- The l2,1-norm objective introduces noise suppression into text-based zero-shot learning and produces classifiers robust against textual noise.The method simultaneously suppresses noisy text signals and learns a function matching visual and textual domains.
- Textual noise significantly affects zero-shot learning performance, as methods without noise suppression perform worse than the proposed approach.
- Most noisy components are suppressed rather than eliminated, preserving combinations of individually unimportant words that can provide discriminative power.
- Useful denoised representation contains attribute-like words, weakly category-related words, and human-uninformative words with distributional patterns across related categories.
- The method improves automatic zero-shot learning by suppressing words with little or no value and addressing variations in human expression.