PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Cue Zero-Shot Learning with Strong Supervision

Zeynep Akata, Mateusz Malinowski, Mario Fritz, Bernt Schiele

arXiv:1603.08754v1cs.CV

TL;DR

Zero-shot recognition must classify unseen visual classes without examples, yet strong existing methods depend on costly human attribute annotations. The paper combines multiple text sources with semantic visual-part supervision in a joint embedding framework. It reports improved recognition and retrieval, including 56.5% versus 50.2% on CUB in the supervised setting.

  • Problem

    Zero-shot learning seeks to recognize novel classes without training examples, but leading methods rely on human attribute annotations that are not readily available at scale.

  • Method

    The framework jointly embeds multiple language parts and semantic visual parts, using strong visual supervision to compensate for weaker textual auxiliary information.

  • Results

    The approach improves fine-grained zero-shot recognition and retrieval, reaching 56.5% versus 50.2% on CUB in the supervised setting.

  • Takeaways & Limitations

    Multiple visual parts, multiple language parts, and word2vec-based class-attribute differences together improve fine-grained zero-shot performance with supervised and unsupervised text embeddings.

  • Takeaways & Limitations

    The NAD3 variant assumes knowledge of which attributes are present for each class, while the visual supervision uses annotated part positions.

Abstract

from arXiv · show

Scaling up visual category recognition to large numbers of classes remains challenging. A promising research direction is zero-shot learning, which does not require any training data to recognize new classes, but rather relies on some form of auxiliary information describing the new classes. Ultimately, this may allow to use textbook knowledge that humans employ to learn about new classes by transferring knowledge from classes they know well. The most successful zero-shot learning approaches currently require a particular type of auxiliary information -- namely attribute annotations performed by humans -- that is not readily available for most classes. Our goal is to circumvent this bottleneck by substituting such annotations by extracting multiple pieces of information from multiple unstructured text sources readily available on the web. To compensate for the weaker form of auxiliary information, we incorporate stronger supervision in the form of semantic part annotations on the classes from which we transfer knowledge. We achieve our goal by a joint embedding framework that maps multiple text parts as well as multiple semantic parts into a common space. Our results consistently and significantly improve on the state-of-the-art in zero-short recognition and retrieval.

1. Introduction

Zero-shot learning transfers auxiliary knowledge to recognize novel visual classes without training examples, but human attribute annotations are costly and scarce. The paper replaces them with multiple web text cues and compensates with strong visual-part supervision in a joint embedding framework.

  • 1. Introduction: Zero-shot learning recognizes novel classes without training examples by transferring auxiliary information, but human attribute annotations are not readily available at scale.The paper motivates using textbook-like and online textual resources instead.
  • 1. Introduction: Stronger semantic-part supervision is used to compensate for weaker but more broadly available language information.The motivation includes detailed part annotations for fine-grained recognition and related datasets with key-point annotations.
  • 1. Introduction: The framework jointly embeds multiple language cues and semantic visual parts, integrating diverse textual and visual sources in a shared space.It adapts Deep Fragment Embeddings to zero-shot learning and supports both unstructured text and attributes.
  • 1. Introduction: The paper reports improved fine-grained zero-shot learning using both unsupervised text sources and supervised attribute annotations when available.The framework is evaluated as a way to improve the state of the art under different auxiliary-information settings.

2. Related Work

Zero-shot recognition requires side information linking disjoint training and test classes. Prior work commonly uses manually collected attributes, whose cost grows substantially for fine-grained categories, while other work explores automatically collected side information and joint vision-language embeddings.

  • 2. Related Work: Because training and test classes are disjoint, zero-shot recognition requires side information modeling relationships between classes.Traditional supervised methods cannot directly use per-image class labels for unseen test classes.
  • 2. Related Work: Human-interpretable attributes have produced promising results, but collecting enough fine-grained attributes is costly and may require expert recognition of subtle differences.Fine-grained datasets often need many attributes, including traits distinguishable only by field experts.
  • 2. Related Work: Related work also collects side information automatically and embeds visual and textual representations jointly for tasks including retrieval, language generation, and image question answering.The paper distinguishes its approach by integrating a wider range of textual representations and strong visual supervision.

3. Zero-Shot Multi-Cue Embeddings

The paper formulates zero-shot recognition as compatibility learning between images and textual class descriptions, combining multiple language and visual parts in a shared embedding space. Its margin-based objective aligns relevant parts while using stronger visual supervision to make noisy, automatically obtained text more useful.

  • 3. Zero-Shot Multi-Cue Embeddings: Zero-shot classification is formulated as learning compatibility between images and textual descriptions, replacing reliance on human attribute input with broader textual sources.Automatically extracted text had previously produced accuracy drops, motivating a richer multi-cue formulation.
  • 3. Zero-Shot Multi-Cue Embeddings: The model maps sets of visual parts and language parts into a common embedding space and predicts the class with maximum compatibility.Language cues include human attributes, word2vec, and bag-of-words representations.
  • 3. Zero-Shot Multi-Cue Embeddings: The compatibility function aggregates interactions between the visual-part set for an image and the language-part set describing a class.This supports multiple cues on both sides rather than requiring a single visual or textual representation.
  • 3. Zero-Shot Multi-Cue Embeddings: The objective combines the part-alignment loss with parameter regularization over the language and visual encoders.The framework parameters are the language and visual embedding matrices.
  • 3. Zero-Shot Multi-Cue Embeddings: Training uses a margin objective that scores true image-class pairs higher than alternatives, with a part-alignment term rewarding relevant language parts.The hinge-style maximum is truncated at zero for assignments already satisfying the margin.

4. Semantic Visual Parts

Semantic parts provide strong visual supervision for fine-grained zero-shot learning, where shared object parts can distinguish categories. The method uses annotated part positions without requiring names or one-to-one correspondence between textual and visual parts.

  • 4. Semantic Visual Parts: Semantic part annotations are hypothesized to improve fine-grained zero-shot learning because discriminative parts help distinguish categories sharing object structure.The paper illustrates this with species identified by combinations of visual parts such as feet and beak shape.
  • 4. Semantic Visual Parts: The method uses part positions but does not require part names, part types, or one-to-one correspondence between textual and visual parts.This allows unsupervised language representations to be paired with strongly supervised visual information.
  • 4. Semantic Visual Parts: A pretrained CNN extracts multiple visual-part descriptors from annotated bounding boxes covering the whole image and locations such as the head and body.The implementation uses 19 bounding boxes, including 15 expert-annotated part locations.

5. Language Parts

The paper develops language-part representations for zero-shot learning that replace costly human attribute annotations with multiple cues extracted from word vectors and Wikipedia text. These include NAD variants based on word2vec class–attribute relationships and MBoW variants that preserve different document structures.

  • 5. Language Parts: Multiple language representations are investigated to transfer class knowledge from less explicit textual sources such as Wikipedia, including attributes, word2vec, BoW, NAD, and MBoW.The framework is designed to accommodate diverse textual cues in an unsupervised setting.
  • 5.2. NAD: Noun-Attribute-Differences: NAD uses word2vec vector differences to estimate class–attribute similarity without directly asking humans to judge each relationship.Three variants differ in whether they use all attributes, automatically selected relevant attributes, or known class–attribute presence.
  • 5.2. NAD: Noun-Attribute-Differences: NAD1 produces one language part per class, whereas NAD2 and NAD3 extend the representation using class-specific attribute information.NAD2 selects top-n nearest attributes in word2vec space, while NAD3 additionally assumes knowledge of which attributes are present for each class.
  • 5.2. NAD: Noun-Attribute-Differences: NAD3 assumes attribute presence for each class but does not require the importance or strength of each attribute.This assumption distinguishes NAD3 from the variants intended to avoid class-specific attribute annotations entirely.
  • 5.3. Multiple Bag-of-Words: MBoW creates multiple bag-of-words representations by using whole articles, fixed paragraph partitions, or Wikipedia’s subject-separated sections.MBoW1 has one language part, MBoW2 uses a fixed number of paragraphs, and MBoW3 produces a variable number of vectors according to article structure.

6. Experiments

Experiments on CUB evaluate joint embeddings across visual and language parts for zero-shot classification and retrieval. Multiple visual parts and carefully chosen multiple language parts improve performance, with the strongest results from their combination.

  • 6.2. Strong Supervision by Part Annotations: 56.5% accuracy with multiple visual parts improves supervised zero-shot classification over 47.0% with training-only parts and 43.3% with a single part.For word2vec, multiple visual parts reach 32.1%; Bag-of-Words reaches 26.0%.
  • 6.2. Strong Supervision by Part Annotations: 30.7% mAUC with multiple visual parts and word2vec exceeds 22.8% with one visual part and 13.0% prior unsupervised state of the art.Attribute supervision reaches 46.6% mAUC.
  • 6.3. Using Multiple Language Parts: NAD2 with 50 language parts reaches 33.9%, while MBoW variants remain below NAD2 and MBoW3 is less reliable because paragraph lengths vary.NAD3 is weakened by class imbalance in descriptive attributes; MBoW3 relies on variable-length Wikipedia paragraphs.
  • 6.3. Using Multiple Language Parts: Multiple language parts help only with multiple visual parts: single visual parts score 25.0% with one language part but 23.6% with multiple language parts.The summary table identifies visual-part support as necessary for the benefit of multiple language parts.
  • 6.3. Using Multiple Language Parts: 33.9% accuracy with multiple visual and language parts establishes a new unsupervised-text state of the art on zero-shot classification.Using multiple visual parts at training and testing with multiple language parts achieves 33.9% accuracy.
  • 6.3. Using Multiple Language Parts: Combining word2vec and BoW reaches 34.7% with multiple visual parts at test time, while combining NAD2 and BoW reaches 34.3%.Both combinations outperform their individual representations in the reported settings.

7. Conclusion

The paper extends fine-grained zero-shot learning by combining diverse language cues with detailed visual part annotations, improving results in supervised and unsupervised settings.

  • 34.7% accuracy improves the unsupervised state of the art by combining different unsupervised text embeddings.The unsupervised result rises from 24.2% to 34.7%.
  • 56.5% accuracy improves the supervised CUB state of the art from 50.2% using strong visual supervision and human-annotated attributes.
  • The formulation integrates diverse class descriptions and detailed part annotations through a joint embedding of language and visual information.
  • Multiple visual parts, multiple language parts, and word2vec distances between class and attribute names are identified as useful extensions for fine-grained zero-shot learning.
Loading 1603.08754v1…