Source-linked AI summary
Open Vocabulary Scene Parsing
Hang Zhao, Xavier Puig, Bolei Zhou, Sanja Fidler, Antonio Torralba
TL;DR
Open-vocabulary scene parsing remains difficult because recognition models and datasets use limited category sets, while evaluation for open-ended recognition is not widely accepted. The paper embeds image pixels and word concepts in a shared space structured by semantic relations, evaluates open-vocabulary predictions on ADE20K, and explores properties of the learned space.
Problem
Recognition models and datasets have limited category vocabularies, and there is no widely accepted way to evaluate open-ended recognition tasks.
Source: §1 · p. 1Method
The framework jointly embeds image pixels and word concepts while preserving WordNet hypernym/hyponym relations for structured open-vocabulary parsing.
Source: §1 · p. 1Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Recognizing arbitrary objects in the wild has been a challenging problem due to the limitations of existing classification models and datasets. In this paper, we propose a new task that aims at parsing scenes with a large and open vocabulary, and several evaluation metrics are explored for this problem. Our proposed approach to this problem is a joint image pixel and word concept embeddings framework, where word concepts are connected by semantic relations. We validate the open vocabulary prediction ability of our framework on ADE20K dataset which covers a wide variety of scenes and objects. We further explore the trained joint embedding space to show its interpretability.
1. Introduction
The paper frames scene parsing as open-vocabulary recognition, addressing fixed category sets and the lack of widely accepted evaluation for open-ended recognition. It proposes pixel-and-concept embeddings that use semantic hierarchy to support structured predictions and evaluates the framework on ADE20K.
1. Introduction
- The framework aims to segment image objects with open-vocabulary labels, using taxonomic fallback when uncertain to make plausible predictions for categories unseen during training.For example, it may predict “vehicle” when it has not seen “tricycle.”Source: §1 · p. 1
- The method embeds image pixels and word concepts in a shared space while preserving WordNet hypernym/hyponym relations, enabling structured predictions and extensible vocabulary.The authors also say this structure reduces the effect of inconsistent annotations from workers with different domain knowledge.Source: §1 · pp. 1–2
- The work explores evaluation metrics for open-vocabulary parsing and other large-scale recognition tasks where category confusions occur.The framework’s open-vocabulary prediction ability is evaluated on ADE20K, and the embedding space is explored through retrieval, boundary loosening, and concept synthesis with arithmetic.Source: §1 · p. 2
1.1. Related work
- Existing parsing models perform well on datasets with limited category sets but are not easily adapted to new objects, motivating predictions beyond those fixed categories.The paper notes benchmarks spanning 20, 30, and 150 categories.Source: §1.1 · p. 2
- The authors distinguish their approach by encoding asymmetric semantic hierarchy in an end-to-end trainable network, which they describe as scalable to larger datasets.They contrast it with prior approaches combining deep networks and optimization methods such as conditional random fields.Source: §1.1 · p. 2
2. Learning joint embeddings for pixel features and word concepts
The framework treats scene parsing as pixel-level retrieval in a shared space for image features and word concepts. Its two streams preserve concept hierarchy and align image pixels with labels, with their losses combined for training.
2. Learning joint embeddings for pixel features and word concepts
- Image features should lie near their concept labels, while concept vectors preserve hypernym/hyponym relations; in this space, coordinate-wise ordering encodes the hierarchy.Vectors near the origin represent general concepts, while larger norms represent greater specificity.Source: §2 · p. 2
2.1. Scoring functions
- The scoring-function section contrasts symmetric Lp and cosine scores with an asymmetric hypernym score, whose value changes when its arguments are swapped.The hypernym score is designed to reveal asymmetric relations between concepts.Source: §2.1 · p. 3
2.2. Concept stream
- The concept stream maps WordNet concepts into embeddings and scores whether one concept is a hypernym of another, training the mapping with a max-margin loss.Ancestor–descendant pairs are positive examples; negative pairs include inverted or random pairs, and the DAG root “entity” is fixed at the origin.Source: §2.2 · p. 3
2.3. Image stream
- The image stream uses a fully convolutional network and a lookup layer shared with the concept stream to embed pixel features alongside their labels.The experiment uses a softmax loss, which the authors report empirically performed better than the described ranking-loss approach.Source: §2.3 · pp. 3–4
2.4. Joint model
- The joint model combines image and concept-stream losses as L = Limage + λLconcept, with λ = 5 and embedding dimension N = 300.The combined objective is intended to preserve both concept hierarchy and visual-feature similarity.Source: §2.4 · p. 4
3. Evaluation Criteria
The section introduces evaluation criteria for open-vocabulary parsing, where metrics should account for semantic distance in the concept graph and the dataset’s highly unbalanced class distribution.
3.1. Baseline flat metrics
- Four flat criteria measure parsing over limited classes: pixel-wise accuracy, mean accuracy, mean IoU, and weighted IoU.Weighted IoU weights intersection-over-union by each class’s pixel ratio.Source: §3.1 · p. 4
3.2. Open vocabulary metrics
- Open-vocabulary metrics should reflect graph distance between prediction and ground truth while representing highly unbalanced class distributions.The section explores metrics for parsing tasks beyond the limited-class setting.Source: §3.2 · p. 4, §3 · p. 4
3.2.1 Hierarchical precision, recall and F-score
- Hierarchical precision, recall, and F-score use shared hypernyms in the vocabulary hierarchy, with the lowest common ancestor capturing their shared depth.For guitar predicted as piano, hierarchical F-score is 0.8; predicting the shared parent musical instrument scores 0.89.Source: §3.2.1 · p. 4
3.2.2 Information content ratio
- Information content ratio scores a prediction using the information content of the lowest common ancestor and the ground-truth and predicted concepts.Concept frequency includes its own pixel frequency and those of its descendants; the metric is intended to address frequent-class dominance and reward inference difficulty and hierarchical accuracy.Source: §3.2.2 · pp. 4–5
4. Experiments
The experiments associate ADE20K classes with WordNet concepts and train concept and image streams for joint embeddings, with distinct training and zero-shot inference procedures.
4.1. Image label and concept association
- AMT annotation matched 3,019 ADE20K classes to WordNet synsets, yielding 2,019 unique synsets in a DAG with an average of 8.2 synsets between matched concepts and the root.Annotation depths range from 4 to 19.Source: §4.1 · p. 5
4.2.1 Concept stream
- The concept stream trains on positive hypernym pairs from graph transitive closure and randomly generated negative pairs; the authors report that closure pairs improve embedding performance.Examples of positive pairs include neckwear–tie, clothing–tie, and entity–tie.Source: §4.2.1 · p. 5
4.2.2 Image stream
- The image stream adapts VGG-16 with dilated convolution layers and embeds 4,096-dimensional pixel features into a positive 300-dimensional space.Fixing pixel-embedding norms at 30 stabilizes training and speeds convergence.Source: §4.2.2 · p. 5
4.2.3 Training and inference
- Training first learns concept embeddings, then uses them to initialize joint training; inference selects among 150 training labels or thresholds scores for zero-shot predictions.The zero-shot threshold is selected using 100 validation images; VGG-ImageNet initializes the image stream and Adam uses a 1e-3 learning rate.Source: §4.2.3 · pp. 5–6
4.3. Results on SceneParse150
On ADE20K, the proposed models are competitive on the 150 training classes and perform strongly on zero-shot parsing, where Joint-Hyper leads under hierarchical metrics. The results also show that embedding norms matter for hypernym predictions and that the model can make plausible, if sometimes noisy, predictions.
4.3. Results on SceneParse150
- Training uses the 150 most frequent ADE20K stuff and object classes, each covering at least 0.02% of dataset pixels; the compared models share a convolutional architecture.Source: §4.3 · p. 6
4.3.1 On the training classes
- On the 150 training classes, Softmax performs best; Joint-Cosine and Word2Vec+ fall short by only around 1%.The authors describe this gap as an affordable sacrifice given zero-shot prediction capability and interpretability.Source: §4.3.1 · p. 6
4.3.2 Zero-shot predictions
- Joint-Hyper performs best under hierarchical metrics on zero-shot predictions, and the proposed models generally beat the baselines by a large margin.The evaluation covers 500 less frequent object classes and allows predictions of their hypernyms.Source: Table 2 · p. 7, §4.3.2 · p. 7
- The Convex Combination baseline uses Softmax outputs to form word-space embeddings and retrieve nearest neighbors without retraining, while still offering reasonable performance.Softmax and Conditional Softmax cannot infer outside the 150 training classes, so their predictions are evaluated within those classes.Source: §4.3.2 · pp. 6–7
- Most proposed models retrieve testing-class hypernyms, but *-Cosine does not because its scoring discards norm information needed for hypernym predictions.Source: §4.3.2 · p. 7
- Joint-Hyper sometimes gives multiple, noisy predictions for one object, including plausible specific and general labels across different parts of a rocking chair.The upper part is labeled “chair,” while the lower part is labeled “furniture.”Source: §4.3.2 · p. 7
4.4. Diversity test
The diversity test asks how many training classes are needed for zero-shot generalization. With Joint-Hyper, performance appears to saturate after training on more than 500 classes, and the authors suggest that more high-quality data may be needed to improve it.
4.4. Diversity test
- The experiment probes how many training classes are needed to generalize well on zero-shot tasks.Source: §4.4 · p. 7
- The test uniformly samples training and testing classes from the pixel-count histogram, fixes the zero-shot test set at 500 classes, and varies training classes from 50 to 1500.Training losses are weighted by information content to offset pixel-number imbalance.Source: §4.4 · pp. 7–8
- Joint-Hyper performance appears to saturate after training with more than 500 classes in the diversity test.The authors conjecture that many classes with few instances may introduce sample noise and suggest that more high-quality data is needed to further improve performance.Source: §4.4 · p. 8
5. Interpreting the embedding space
The section probes how the joint embedding space supports concept search, implicit attribute detection, and arithmetic synthesis, with hierarchical information shaping its behavior.
5. Interpreting the embedding space
- Elementwise max and min synthesize concepts: max acts as intersection, while min acts as union.Examples are pool table as the common hyponym of “table” and “game equipment,” and cart as combining attributes of “bicycle” and “canopy.”Source: Figure 9 · p. 8, §5 · p. 8
- For increasingly abstract search concepts, the joint embedding space far outperforms Word2Vec+.The framework can search with WordNet synsets that were not trained with image data because semantics propagate through joint training.Source: §5 · p. 8
- Loosening the “chair” threshold also gives reasonable responses for non-hyponyms “bench” and “ottoman,” suggesting implicit affordance attributes such as “sittable.”The authors conjecture that the embedding space clusters objects by abstract attributes.Source: §5 · p. 8
1. Data association protocol
ADE20K labels are associated with WordNet synsets through a protocol that handles unambiguous, ambiguous, and unmatched class names.
1. Data association protocol
- The protocol first searches WordNet for synsets sharing each ADE20K class name, yielding single-match, multiple-match, or no-match cases.The no-match case can arise when the concept has another name or is absent from WordNet.Source: §1 · p. 10
- For a single match, workers verify the automatic class–synset pairing using image labels and WordNet definitions.Source: §1 · p. 10
- For multiple or absent candidates, workers choose a matching synset from definitions or search WordNet online, with a no-match option.Multiple candidates are presented with an image; for absent candidates, workers inspect an image and use the WordNet online API.Source: §1 · p. 10
2. Concept graph
The ADE20K-to-WordNet association yields a hierarchical concept graph with thousands of matched classes and synsets.
2. Concept graph
- 3,019 ADE20K classes map to 2,019 unique synsets forming a DAG, with entity.n.01 as top hypernym and an average of 8.2 synsets between.Annotation depths range from 4 to 19.Source: §2 · p. 10
- Figure 11 visualizes only part of the graph: node radii encode ADE20K class frequencies, and nodes with five or fewer descendants are hidden.Source: §2 · p. 10
3. Full zero-shot prediction lists
The zero-shot results show that prediction specificity varies with the example: the model is conservative on hard cases and more specific on easy ones.
3. Full zero-shot prediction lists
- The zero-shot prediction lists are hierarchical: hard “dome” examples receive general, conservative predictions, while easy “wagon” examples receive specific, aggressive ones.Figure 12 marks correct predictions in green and inconsistent items in orange.Source: §3 · p. 10, Figure 12 · p. 13