PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-vocabulary Attribute Detection

María A. Bravo, Sudhanshu Mittal, Simon Ging, Thomas Brox

arXiv:2211.12914v2cs.CVcs.LG

TL;DR

Open-vocabulary research has largely focused on object classes, leaving object-level attribute detection without a reliable evaluation benchmark. The paper introduces the OVAD task and a densely annotated benchmark, and reports that foundation models still perform clearly worse on attributes than on objects.

  • Problem

    Open-vocabulary methods focus on object classes, and it is unclear how well they generalize to object-level attributes.

    Source: §1 · p. 1

  • Method

    The paper defines OVAD and builds a clean, densely annotated benchmark with positive and negative labels, alongside a baseline and evaluations of foundation models.

    Source: §1 · pp. 1–2

  • Results

    Foundation-model performance on attributes remains clearly behind performance on objects, while the caption-informed baseline outperforms open-vocabulary detection models for OVAD.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The benchmark provides a basis for evaluating open-vocabulary attribute methods and foundation models.

    Source: §1 · p. 2

    Test attributes must not be used to build the model; their set is revealed only at test time.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Vision-language modeling has enabled open-vocabulary tasks where predictions can be queried using any text prompt in a zero-shot manner. Existing open-vocabulary tasks focus on object classes, whereas research on object attributes is limited due to the lack of a reliable attribute-focused evaluation benchmark. This paper introduces the Open-Vocabulary Attribute Detection (OVAD) task and the corresponding OVAD benchmark. The objective of the novel task and benchmark is to probe object-level attribute information learned by vision-language models. To this end, we created a clean and densely annotated test set covering 117 attribute classes on the 80 object classes of MS COCO. It includes positive and negative annotations, which enables open-vocabulary evaluation. Overall, the benchmark consists of 1.4 million annotations. For reference, we provide a first baseline method for open-vocabulary attribute detection. Moreover, we demonstrate the benchmark's value by studying the attribute detection performance of several foundation models. Project page https://ovad-benchmark.github.io

1. Introduction

Open-vocabulary detection extends object recognition beyond fixed training classes, but it remains unclear how well vision-language models represent object attributes. The paper defines OVAD and introduces a densely annotated benchmark for testing attribute detection alongside objects.

1. Introduction

  • Existing open-vocabulary methods use image-text pairs to learn novel objects, but it is unclear how well they generalize beyond object classes to attributes.The paper focuses on object-level attributes such as state, size, and color.Source: §1 · p. 1
  • OVAD queries open sets of objects and their attributes through text prompts at inference, without knowledge of the tested classes during training.The task has two stages: open-vocabulary object detection, then predicting attributes for each detected object; all attributes are novel.Source: §1 · pp. 1–2
  • The OVAD benchmark covers 117 attribute classes across 80 MS COCO object classes, with 1.4 million annotations and positive and negative labels.It averages 98 attribute annotations per object instance and 7.2 objects per image.Source: §1 · p. 2
  • The benchmark uses manually annotated, quality-tested labels and negative attribute annotations to support evaluation of misses and false positives.It covers visually identifiable attributes and extends the OVD benchmark to all 80 COCO object classes, increasing novel object classes from 17 to 32.Source: §1 · p. 2
  • The paper evaluates several open-source vision-language models to assess how well their object-class success generalizes to attributes.The tested models include CLIP, Open CLIP, BLIP, ALBEF, and X-VLM.Source: §1 · p. 2
  • The authors provide an attribute-focused baseline that outperforms open-vocabulary models aimed only at object classes.They also test several open-source foundation models on visual attribute detection.Source: §1 · p. 2

2. Related Work

Prior work addresses fine-grained attributes, but existing benchmarks vary in scope, density, and suitability for isolating attribute performance. This paper extends open-vocabulary detection to attributes and evaluates vision-language models on that task.

2. Related Work

  • Prior attribute research includes part-based descriptions, narrow-domain fine-grained benchmarks, and zero-shot methods that use attributes as intermediate or compositional signals.This work instead evaluates zero-shot detection and discrimination of object-level attributes.Source: §2 · p. 2
  • COCO Attributes covers 29 object categories, while Visual Genome has over 68,000 attribute categories but only 0.74 annotations per object instance.Visual Genome’s sparse, noisy annotations and lack of negatives make it unsuitable for OVAD evaluation.Source: §2 · p. 2
  • VAW provides positive and negative labels across many attribute and object categories, but automated filtering leaves annotations sparse per image and object.Open Images covers 288 object categories but only 15 attribute categories, which are not densely annotated.Source: §2 · pp. 2–3
  • Visual question answering datasets include questions about attributes, but their scores do not isolate attribute performance from noun recognition..Source: §2 · p. 2
  • Open-vocabulary detection work learns novel object classes from image-caption pairs and has been extended with localization and training improvements; this paper expands the formulation to attributes.Earlier work also demonstrated qualitative fine-grained detection by adding adjectives to object queries.Source: §2 · p. 3
  • The paper challenges five state-of-the-art vision-language models on the fine-grained task of open-vocabulary attribute detection.These models align visual concepts with language representations using cross-modal objectives.Source: §2 · p. 3

3. Open-vocabulary Attribute Detection

OVAD combines open-vocabulary object detection with attribute prediction, where all attribute classes are novel at training time. Its benchmark pairs densely annotated MS COCO objects with visually identifiable attributes and supports evaluation in detection and box-oracle settings.

3.1. The OVAD Task

  • OVAD evaluates open-vocabulary object detection and attribute presence or absence for each detected object, with attributes unavailable during training and revealed only at test time.The box-oracle setting provides ground-truth boxes and object classes at inference, isolating attribute detection from object-detection errors.Source: §3.1 · pp. 3–4

3.2. The OVAD Benchmark

  • The benchmark is a test set, not a supervised-training or validation set, and was built from 2,000 MS COCO validation images with revised object boxes.The resulting 14,300 object instances were manually labeled under strict guidelines.Source: §3.2 · p. 4
  • The 117 attributes were selected from COCO captions and filtered to retain visually identifiable categories with positive examples among the 80 COCO object classes.The attributes are organized into 19 types, including color, pattern, material, maturity, and cooking state.Source: §3.2 · p. 4
  • Human annotators label attributes as positive, negative, or unknown, marking non-selected attributes within an exclusive type as negative or unknown as appropriate.Unknown labels are excluded from evaluation; infeasible object-attribute combinations are manually excluded.Source: §3.2 · pp. 4–5
  • The benchmark contains 1,401,484 attribute annotations across 2,000 images, 117 attribute categories, 80 object categories, and 14,300 object instances.It includes 122,998 positive and 1,278,486 negative labels, plus 172,760 unknown labels; each box has an average of 96.8 attribute annotations.Source: §3.2 · p. 5
  • Attribute mAP is averaged across categories; open-vocabulary detection matches objects at IoU ≥0.5, while box-oracle evaluation uses ground-truth boxes.The benchmark also reports OVD-80 object-detection AP50; attributes of unmatched ground-truth objects are marked absent.Source: §3.2 · p. 5

4. OVAD Baseline Method

OVAD-Baseline aligns a Faster R-CNN detector’s region embeddings with a frozen CLIP language space. It trains with object detection, image-caption matching, and caption-derived proxy labels, then scores queried object and attribute classes by region-text similarity.

4. OVAD Baseline Method

  • OVAD-Baseline combines a frozen language model with a Faster R-CNN detector whose classification head projects visual features into the language embedding space.The baseline uses CLIP as its language model.Source: §4 · p. 5
  • The detector is trained to localize base objects, match images with captions, and use caption concepts as proxy labels for novel classes and attributes.Its detection loss combines region-proposal, box-regression, and similarity-based classification losses; image-caption pairs use a contrastive matching loss.Source: §4 · p. 6
  • Caption nouns, noun phrases, and noun complements serve as proxy labels matched to image regions, with noun complements providing more explicit attribute supervision.For concepts whose locations are unknown, proxy labels are matched to the largest predicted box; arbitrary image-caption pairs provide negatives.Source: §4 · p. 6
  • At inference, region embeddings are compared with text embeddings for object and attribute classes, and sigmoid similarities yield separate presence predictions.The class embeddings average representations of all class synonyms without a special prompt.Source: §4 · p. 6, Figure 3 · p. 6

5. Experiments

Experiments compare the OVAD baseline with open-vocabulary detectors and evaluate foundation models on attribute recognition. The baseline outperforms the compared detectors, while foundation-model results show that attribute recognition remains challenging and responds to data and alignment choices.

5.1. Open-vocabulary Attribute Detection

  • The OVAD-Baseline outperforms the compared open-vocabulary detection methods on attribute detection, and methods strong on object detection are not necessarily strongest on OVAD.All methods exceed chance; region-to-text matching methods achieve better attribute mAP than methods matching an image to a single text representation.Source: §5.1 · p. 7
  • Parts-of-caption proxy-labels improve both object and attribute detection, while noun complements make attribute supervision more explicit.The ablation identifies caption segmentation and compositional language structure as useful design choices for the baseline.Source: §5.1 · p. 7

5.2. Foundation Models Applied to Attributes

  • Foundation models show low absolute attribute-detection performance; a supervised detector trained by cross-validation on the evaluation dataset reaches 48.16±0.52 mAP.The authors interpret the results as suggesting a bias toward object classes over fine-grained attributes.Source: §5.2 · p. 8
  • 6.25% (OpenCLIP, All attribute performance), 7.06% (BLIP, All attribute performance), and 8.46% (X-VLM, All attribute performance) are the reported gains from scaling training data.ALBEF and BLIP improve All attribute performance by 37.25% and 33.52%, respectively, after further training on curated data with ITC and ITM objectives.Source: §5.2 · p. 8
  • Localized image region-text matching methods perform strongly on attributes, but X-VLM loses 6.76% when fine-tuned for image-caption retrieval.OVAD-Baseline-Box also outperforms foundation models trained on datasets reported as more than 3000 times larger (CLIP and OpenCLIP) and more than 1000 times larger (pretrained ALBEF and BLIP).Source: §5.2 · p. 8

6. Conclusion

The paper introduces OVAD and its benchmark, provides a baseline, and reports that foundation models lag on attributes relative to objects. The supplementary material details the benchmark’s attribute taxonomy and frequency-based class partitions.

6. Conclusion

  • The paper introduces the OVAD task and benchmark, provides a baseline that outperforms OVD models, and finds foundation-model attribute performance clearly behind object performance.The benchmark is described as clean and densely annotated, and the experiments evaluate publicly available foundation models.Source: §6 · p. 8

A.1. Attribute taxonomy

  • The taxonomy groups 117 distinct attributes into 19 types, with human-specific interpretations for color, pattern, and length.The taxonomy also assigns sitting to position and bald to hair length.Source: §A.1 · p. 12

A.2. Attribute distribution

  • The frequency partition contains 15 head, 53 medium, and 49 tail attribute classes, defined using positive-annotation frequency thresholds.Head classes lie above thigh, tail classes below tlow, and medium classes between them.Source: §A.2 · p. 12

A.3. Dataset size analysis

The dataset-size analysis measures variability in OVAD-Baseline-Box mAP across randomly selected subsets of OVAD images. Variability falls as subset size grows, with the tail partition showing similar behavior to head and medium.

A.3. Dataset size analysis

  • At 23% of the dataset, mAP standard deviation is below 1% for all attribute partitions, and tail variability is similar to head and medium.The analysis averages results across six experiments with different data shuffles, using ground-truth boxes and non-overlapping image subsets.Source: §A.3 · p. 12
  • The analysis uses frequency-defined all, head, medium, and tail partitions and subset sizes ranging from 3% to 33% of OVAD images.The maximum subset size allows at least three non-overlapping image sets for standard-deviation calculation.Source: §A.3 · p. 12
  • The estimated mAP standard deviation for tail attributes is below 0.3% when extrapolating to the full 2,000-image dataset.This value comes from extrapolating the tail curve to 100% of dataset size.Source: §A.3 · p. 13

B. Dataset Creation

The benchmark was built from human annotations with structured label rules and iterative quality checks. Its restricted attribute options can miss unusual object–attribute combinations, adding noise to the annotations.

B.1. Annotation process

  • 2,000 MS COCO validation images were selected, and object annotations provided the starting point for a fresh attribute annotation process guided by 19 attribute types and the taxonomy.For each object, annotators selected attributes as positive or unknown; the system used exclusivity rules to derive negative or unknown labels, with separate handling for color and state.Source: §B.1 · p. 13

B.2. Annotation quality control

  • Approximately 80% of annotations were revised through repeated annotator review, while the remaining 20% received only one annotation round.Independent checks found 89.44% consistency for revised images, 86.35% for non-revised images, and 91.26%±2.79 for a 50-image golden set.Source: §B.2 · p. 14

B.3. Human bias in attribute annotation

  • Fourteen annotators from six nationalities and varied age groups, sexes, and skill levels contributed to the annotations, which the authors say balances cultural, age, and sex biases.Annotators spent an average of 12 minutes per image; revisers spent about 3 minutes per image.Source: §B.3 · pp. 14–15

B.4. Exceptions in attribute annotation

  • Options restricted by object category can miss exceptional attributes, including cases where an attribute is visually indiscernible or an unusual object–attribute pairing occurs.Figure 7 illustrates missed cases as well as exceptions that the system can represent, such as material for animals or vehicles.Source: §B.4 · p. 15

B.5. Dataset annotations and visualization

  • The supplementary JSON uses MS COCO-compatible annotations, with each instance’s “att vec” encoding positive, negative, and unknown labels as 1, 0, and -1.The supplementary material also includes an attribute list and a visualization demo with object-and-attribute search.Source: §B.5 · p. 15

C.1. OVAD supervised training ablation

A supervised box-oracle experiment tests the benchmark’s feasibility, while cross-dataset experiments assess transfer from existing attribute datasets. The supervised model substantially exceeds chance, and transfer models achieve lower reported mAP values.

C.1. OVAD supervised training ablation

  • 48.16±0.52 mAP was achieved by the supervised ResNet50, versus 8.29±0.06 mAP at chance and 23.30±0.76 mAP for OVAD-Baseline -Box on the same splits.The experiment used 4-fold cross-validation with 500 test images per run and 1,500 training images.Source: §C.1 · p. 15

C.2. Cross-dataset transfer ablation

  • 18.20 mAP after training on VAW and 15.96 mAP after training on COCO Attributes were achieved on OVAD in the box-oracle setting.Both models were trained on cropped objects and used a projection layer to match the CLIP text-encoder representation dimension.Source: §C.2 · p. 15

D.1. Implementation details

The baseline combines a ResNet50 visual detector with CLIP text embeddings, using visual–text similarity for attribute classification. Training uses caption and box-label supervision, while inference averages synonym representations for each attribute.

D.1. Implementation details

  • The baseline uses an ImageNet-pretrained ResNet50 detector and CLIP text encoder, with cosine similarity between an object-box representation and text embedding used for training losses and inference scores.Attribute prediction scores use the same similarity score.Source: §D.1 · pp. 15, 17
  • Training uses a binary cross-entropy objective, offline text representations, caption and parts-of-caption matching, and SGD for 90k iterations with a 1k-step warmup.The model uses τ=50 for object and attribute prediction; training batches alternate between box-plus-class labels and caption-based labels.Source: §D.1 · pp. 15–16
  • At inference, each attribute’s text embedding is the average representation of its synonymous terms.Figure 7 depicts annotation exceptions and corner cases.Source: §D.1 · p. 16, Figure 7 · p. 16

E.1. Open-vocabulary attribute detection results

The evaluation reports a novel-category AP50 advantage for the method alongside a trade-off in base-class performance. Across attribute categories, reported performance varies by attribute type and foundation model.

E.1. Open-vocabulary attribute detection results

  • The method achieves high AP50 for novel categories, with a trade-off in base-class performance across the generalized evaluation setting.The comparison uses OVD and OVD-80 image annotations; OVD-80 has 32 novel object classes versus 17 in OVD.Source: §E.1 · p. 17

E.2. Performance per attributes type

  • Material, optical property, order, size, and texture show larger improvement over chance than other attribute types among the five methods.Figure 8 presents performance by attribute type.Source: §E.2 · p. 17
  • X-VLM outperforms the other five foundation models by a large margin for most attribute types in the box-oracle setting.Cooked, gender, and maturity have higher relative improvement over chance.Source: §E.2 · p. 17

F. Qualitative Results

The qualitative results illustrate OVAD baseline predictions against ground truth, while the accompanying materials specify asset licenses and detailed annotation guidelines. Those guidelines define class-dependent attribute assignment and how annotators handle visual evidence, ambiguity, and unknown labels.

F. Qualitative Results

  • For qualitative examples, the first image is the prediction and the second is ground truth; base and novel entities are blue and red, respectively.The displayed predictions use the maximum-overlap match to the ground-truth box and show the top 200 scores per attribute category; examples are drawn from images with high mAP performance.Source: F. Qualitative Results · p. 17

G. Licences of Assets

  • The CVAT application and code are available through its GitHub repository and website under the MIT license.Source: G. Licences of Assets · p. 17
  • MS COCO detection and caption datasets [10] [32] have multiple listed licenses, including Attribution-NonCommercial-ShareAlike and Attribution License.The passage also lists other license variants, no known copyright restrictions, and United States Government Work.Source: G. Licences of Assets · p. 17
  • The OVAD benchmark annotations and website are licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International.Source: G. Licences of Assets · p. 17

H. Annotation Guidelines

  • Annotation guidelines support consistency and agreement by assigning valid attribute types according to object category and attribute descriptions.The guidelines specify four object-category types and 19 attribute types.Source: H. Annotation Guidelines · pp. 17–18
  • Annotators mark attributes positive from visual evidence, use “unknown” for occlusion or ambiguity, and select the most appropriate feasible category.Source: H. Annotation Guidelines · p. 17
  • Color labels vary by object type: hair and clothes color apply to humans, while the human class has no general color attribute.Gender is marked only when evident, and face expression only when the face is clearly visible.Source: H. Annotation Guidelines · pp. 18–19
  • Maturity labels distinguish adult or old from young or baby for humans and animals.Source: H. Annotation Guidelines · p. 19
Loading 2211.12914v2…