ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models

Chunyuan Li; Haotian Liu; Liunian Harold Li; Pengchuan Zhang; Jyoti Aneja; Jianwei Yang; Ping Jin; Houdong Hu; Zicheng Liu; Yong Jae Lee; Jianfeng Gao

Source-linked AI summary

ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models

Chunyuan Li, Haotian Liu, Liunian Harold Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, Jianfeng Gao

arXiv:2204.08790v6Apr 19, 2022cs.CVcs.CLcs.LG

TL;DR

Evaluating language-augmented visual models remains difficult because public benchmarks and easy-to-use, reproducible evaluation toolkits are limited. ELEVATER addresses this gap with a benchmark, automated toolkit, and efficiency metrics spanning diverse classification and detection tasks, and reports stronger language-augmented adaptation than traditional vision-only methods.

Problem
Public benchmarks and accessible evaluation procedures are limited, making fair and reproducible comparison of language-augmented visual models difficult.
Method
ELEVATER combines 20 image-classification and 35 object-detection datasets with external knowledge, automatic hyper-parameter tuning, adaptation methods, and sample- and parameter-efficiency evaluations.
Results
Language-augmented adaptation methods significantly outperform traditional vision-only adaptation methods, while text and vision consistently improve few-shot performance and external knowledge improves zero-shot and few-shot learning.
Takeaways & Limitations
ELEVATER provides reproducible baselines for studying transferable language-augmented visual models, adaptation efficiency, and external knowledge for task-level transfer.
Takeaways & Limitations
External knowledge involves a coverage-quality trade-off: WordNet is precise but limited in coverage, whereas GPT3 offers full coverage but has difficult-to-assess quality.

Abstract

from arXiv · show

Learning visual representations from natural language supervision has recently shown great promise in a number of pioneering works. In general, these language-augmented visual models demonstrate strong transferability to a variety of datasets and tasks. However, it remains challenging to evaluate the transferablity of these models due to the lack of easy-to-use evaluation toolkits and public benchmarks. To tackle this, we build ELEVATER (Evaluation of Language-augmented Visual Task-level Transfer), the first benchmark and toolkit for evaluating(pre-trained) language-augmented visual models. ELEVATER is composed of three components. (i) Datasets. As downstream evaluation suites, it consists of 20 image classification datasets and 35 object detection datasets, each of which is augmented with external knowledge. (ii) Toolkit. An automatic hyper-parameter tuning toolkit is developed to facilitate model evaluation on downstream tasks. (iii) Metrics. A variety of evaluation metrics are used to measure sample-efficiency (zero-shot and few-shot) and parameter-efficiency (linear probing and full model fine-tuning). ELEVATER is a platform for Computer Vision in the Wild (CVinW), and is publicly released at at https://computer-vision-in-the-wild.github.io/ELEVATER/

1 Introduction

ELEVATER addresses the difficulty of fairly evaluating language-augmented visual models by providing an open-source benchmark and toolkit for rigorous, reproducible task-level transfer evaluation. It combines diverse datasets and knowledge, efficiency metrics, and automated adaptation procedures.

Benchmark: ELEVATER provides the first publicly available benchmark for task-level transferability, covering 20 image-classification and 35 object-detection datasets.Each dataset is augmented with an external knowledge base for language data augmentation.
Metrics: The benchmark measures sample-efficiency through zero-shot, few-shot, and full-shot settings, and parameter-efficiency through linear probing and full fine-tuning.These metrics target the costs of deploying models for real-world applications.
Toolkit: Automatic hyper-parameter tuning reduces human labor and supports fair comparisons among model checkpoints.The toolkit also provides model adaptation strategies for downstream tasks.
Adaptation: Language-augmented adaptation methods significantly outperform traditional vision-only adaptation methods and provide baselines for future methods.The methods are designed for pre-trained language-augmented visual models.
Findings: Empirical findings show that text and vision improve few-shot performance, fine-tuning overtakes linear probing as training data grows, and external knowledge improves zero-shot and few-shot learning.Random linear-head initialization is sub-optimal, while few-shot results exceed zero-shot results in the reported study.

2 Related Work: From Class-level to Task-level Transfer

The paper distinguishes traditional class-level zero-shot learning from task-level transfer across unseen datasets and tasks. ELEVATER emphasizes diverse cross-domain evaluation, external knowledge, and the full spectrum of adaptation data regimes.

From Class-level to Task-level Transfer: Traditional class-level zero-shot learning evaluates generalization to unseen object categories, whereas task-level zero-shot learning evaluates generalization to unseen datasets or tasks.The distinction concerns whether the transfer target is a category or an entire task.
From Class-level to Task-level Transfer: Existing zero-shot benchmarks generally use a single domain and manual class splits, while ELEVATER compares task-level transfer without a train-evaluation category split.Table 1 reports dataset statistics and evaluation settings for the benchmark and existing datasets.
From Class-level to Task-level Transfer: ELEVATER evaluates models pretrained on their own large corpora by measuring zero-shot performance across diverse downstream datasets and domains.This task-level transfer setting is presented as more practical for one-model-for-all applications.
From Class-level to Task-level Transfer: ELEVATER adds external knowledge for each dataset and measures sample-efficiency across zero-shot, few-shot, and full-shot settings.These additions extend evaluation beyond benchmarks typically focused on full-shot settings.

3 Benchmarks

ELEVATER benchmarks task-level transfer across diverse image-classification and object-detection datasets, augmenting each dataset with external knowledge and evaluating adaptation costs across data and parameter budgets.

Datasets: ELEVATER includes 20 image-classification datasets and 35 object-detection datasets as downstream evaluation suites.The benchmark is designed to proxy unseen tasks in the wild using datasets from diverse computer-vision domains.
Language/Knowledge Augmentations: Each dataset is paired with external knowledge sources, including WordNet hierarchies and definitions, Wiktionary definitions, and GPT3-generated knowledge.WordNet offers richer and more precise knowledge but lower coverage, whereas GPT3 provides full coverage with harder-to-assess quality.
Evaluation Tracks: ELEVATER provides academic and industry tracks that respectively limit or permit the scale of pre-training data, while requiring reproducible reporting of pre-training datasets.The academic track restricts data to established public datasets; the industry track has no pre-training-scale limit but excludes benchmark images for zero- and few-shot reporting.
Evaluation Settings: Adaptation cost is represented along orthogonal sample-efficiency and parameter-efficiency dimensions, spanning inexpensive to expensive strategies.The bottom-left of the cost chart represents the least expensive adaptation strategy, while the top-right represents the most expensive.
Evaluation Settings: The benchmark measures sample-efficiency with zero-, few-, and full-shot settings and parameter-efficiency with linear probing and full model fine-tuning.For image classification, N is 0, 5, 20, or 50 per category; for object detection, N is 0, 1, 3, 5, or 10.

4 Toolkits

The toolkit standardizes downstream evaluation through automatic hyper-parameter tuning and adaptation methods that preserve or exploit language representations in pre-trained visual models.

Automatic Evaluation: Automatic hyper-parameter tuning avoids human-in-the-loop selection, reducing labor and enabling fair comparisons between model checkpoints.The implementation uses a simple grid-search pipeline and leaves more sophisticated methods such as BOHB and DEHB for future work.
Model Adaptation: Language-augmented visual models use image and text encoders with projection layers that embed both modalities into a joint space for zero-shot prediction.Category text features are averaged across prompts, and the image is assigned the category with the highest image-text similarity u^⊤v.
Model Adaptation: Random-initialized adaptation adds a randomly initialized linear projection to the pre-trained visual projection.This two-projection strategy is the adaptation method described for the original CLIP setup.
Model Adaptation: Language-initialized adaptation initializes the linear head with text features or with WvV, separating visual and text heads while matching trainable-parameter budgets.The Two-Projection variant initializes W with V, while the One-Projection variant initializes Wm with WvV before adaptation.
Model Adaptation: The authors recommend language-initialized methods because they show superior empirical performance and can incorporate downstream external knowledge into prompts.Knowledge is concatenated with the original language prompt for adaptation.

5 Empirical Results and Findings

ELEVATER evaluates language-augmented visual transfer across image classification and object detection, emphasizing language-aware adaptation, sample efficiency, parameter efficiency, and external knowledge. Experiments show benefits from language initialization and knowledge augmentation, with adaptation trade-offs depending on data availability and task.

The Role of Language for Vision: Language-based initialization significantly outperforms random initialization under both linear probing and fine-tuning.
The Role of Language for Vision: Even at 2 shots, language-initialized adaptation can outperform zero-shot CLIP, while fine-tuning surpasses linear probing beyond 20 shots.With random initialization, fine-tuning performs significantly worse than linear probing in few-shot settings.
Sample Efficiency: CLIP consistently gains 5–10% over supervised ViT in 5-shot image classification, indicating greater sample efficiency in limited-data settings.For CLIP, fine-tuning is worse than linear probing at 5 shots but better beyond 20 shots.
Sample Efficiency: GLIP is more sample-efficient than DyHead for object detection, with a larger gap in fewer-shot settings and a smaller gap in full-shot settings.Unlike image classification, fine-tuning consistently outperforms linear probing for object detection.
Parameter Efficiency: Increasing trainable parameters generally improves performance, but 5-shot CLIP shows a slight drop under full-model fine-tuning.This suggests that data-limited adaptation may benefit from an optimal intermediate parameter count.
Parameter Efficiency: Prompt tuning GLIP outperforms fine-tuning DyHead in the 1-shot setting while training less than 0.1% as many parameters.Prompting also outperforms linear probing for GLIP.
External Knowledge: External knowledge improves zero-shot transfer, with UniCL rising from 27.15 to 29.92–33.93 and GLIP-A from 11.53 to 11.70–13.30.Combining GPT-3 and Wiktionary knowledge gives an additional boost; increasing GPT-3 items consistently helps image classification but not object detection.

6 Conclusions

ELEVATER is presented as a platform for evaluating task-level transfer in language-augmented visual models. Its benchmark, toolkit, and baselines support research on transferable models, adaptation efficiency, and external knowledge.

ELEVATER evaluates language-augmented visual models for task-level transfer using 20 image classification and 35 object detection datasets enriched with external knowledge.
Its open-source toolkit combines automatic hyper-parameter tuning with language-initialized adaptation methods to support easy use and fair comparison.
The benchmark provides strong baselines for research on transferable visual models, sample-efficient and parameter-efficient adaptation, and external knowledge.
Designing general-purpose task-level transferable visual models remains largely unanswered, and ELEVATER is intended to support progress on this challenge.

Supplementary Material for “ELEVATER: A Benchmark and

The supplementary material covers societal impact, related NLP work, dataset and knowledge-source details, toolkit implementation, adaptation gaps, and additional experiments. It also frames Computer Vision in the Wild as broad transfer across concepts and visual domains.

The appendix discusses societal impact, NLP pre-training literature, benchmark statistics and licensing, external knowledge construction, toolkit details, adaptation gaps, and additional model comparisons.
The supplementary experiments provide evidence that external knowledge improves CLIP adaptation.
Task Transfer Cost: Adaptation cost is organized along sample-efficiency and parameter-efficiency dimensions, spanning inexpensive and expensive strategies.
Computer Vision in the Wild: The benchmark distinguishes standard close-set recognition, open-set vocabulary transfer, domain shift, and the broader Computer Vision in the Wild setting.
Computer Vision in the Wild: Computer Vision in the Wild requires strong generalization to both new concepts and new visual distributions across customized downstream tasks.

B.2 Related Works in NLP: Benchmarks, Adaptation, and Knowledge

ELEVATER focuses on task-level transfer for vision encoders across domains, complementing broader vision-language evaluation. It draws on NLP benchmark and knowledge-integration practices while documenting dataset scope and bias considerations.

Related Work and Scope: Unlike class-level zero-shot benchmarks, ELEVATER evaluates task-level transfer across diverse downstream datasets and domains.
Related Work and Scope: The benchmark concentrates on image classification and object detection, complementing work on visual question answering, video-language understanding, and vision-enhanced language encoders.
NLP Connections: ELEVATER follows NLP benchmark practice by using suites of datasets to evaluate transfer and by adding external knowledge sources to downstream datasets.
Benchmark Scope: The benchmark contains 20 image classification datasets and 35 object detection datasets.
Caveat: Because the benchmark uses public datasets and annotations, its systems may inherit data biases that can affect predictions and real-world decisions.

C.2 Visualization Comparison with Established Vision Datasets

ELEVATER compares its diverse vision datasets with established benchmarks and augments category prompts with external knowledge, including GPT-3-generated explanations when curated resources are incomplete.

Dataset comparison: ELEVATER uses 35 object-detection datasets, with Box mAP as the evaluation metric.The datasets are downloaded from Roboflow.
External knowledge: External knowledge includes WordNet, Wiktionary, and GPT-3-generated content, with licensing guidance provided for each source.GPT-3-generated knowledge was approved for release as part of ELEVATER.
External knowledge: GPT-3 is used to provide additional explanations because Wiktionary and WordNet do not fully cover all downstream concepts.The method uses in-context learning with multiple concept–explanation examples sampled from concepts with Wiktionary knowledge.
Prompt construction: Prompts combine dataset-specific category names and templates, optionally followed by selected external knowledge.The toolkit maintains mappings from datasets to category names, templates, and knowledge resources.
Evaluation platform: The benchmark supports public ICinW and ODinW leaderboards for comparing models under efficiency considerations.The evaluation setting separates sample-efficiency from parameter-efficiency.

D.2 A new metric with performance-efficiency trade-off

ELEVATER introduces performance-efficiency to combine prediction accuracy with the number of trainable parameters, while analyzing adaptation choices and model taxonomy. Its experiments emphasize language-based initialization and show that channel batch normalization is preferable to instance-wise normalization in the tested setting.

Metric: Performance-efficiency (PE) combines prediction accuracy with parameter-efficiency in a single metric.The metric is designed for the parameter-efficiency track.
Adaptation analysis: Channel batch normalization performs best, while adding instance-wise ℓ2 normalization does not improve performance.The result suggests that downstream adaptation should not automatically reproduce all CLIP pre-training choices.
Adaptation analysis: Language-augmented initialization improves 5-shot fine-tuning by more than 30% on average and is identified as the most critical alignment component.Other factors, including normalization, batch size, and temperature, have much smaller effects in the reported analysis.
Model taxonomy: The taxonomy first separates language-augmented from language-free pre-training, then divides language-augmented models by external-knowledge use.The taxonomy concerns pre-training and is independent of downstream adaptation.

G.3 Experimental Results of Different Model Checkpoints

ELEVATER evaluates checkpoints under standardized adaptation procedures and reports that CLIP ranks highest overall among the compared models. Language-based initialization consistently outperforms random initialization across the tested adaptation dimensions and datasets.

Checkpoint comparison: CLIP, ViT, DeiT, MoCo-v3, and MAE rank in that descending overall order in the reported checkpoint comparison.The experiments use the same automatic hyper-parameter tuning process without model- or dataset-specific tuning.
Evaluation scope: The evaluated checkpoint sets include multiple language-image models and model scales across Academic and Industry Tracks.The Academic Track includes CLIP, DeCLIP, FILIP, and SLIP; the Industry Track includes DeCLIP, OpenCLIP, and CLIP.
Initialization comparison: Language initialization consistently outperforms random initialization across sample-efficiency, parameter-efficiency, and different datasets.The comparison includes linear probing and fine-tuning scores.

H Benefits of External Knowledge in Model Adaptation

ELEVATER studies external knowledge as a prompting and adaptation resource for language-image models. Adding knowledge improves zero-shot performance across the evaluated datasets and supports selective knowledge-source use during few- and full-shot adaptation.

Zero-shot evaluation: External knowledge consistently improves CLIP on four datasets and yields considerable gains on three others in zero-shot settings.The magnitude of improvement varies between datasets, even though CLIP was pretrained without external knowledge.
Few- and full-shot adaptation: Knowledge-augmented adaptation selects among knowledge sources using split-validation accuracy before testing.This strategy contrasts with knowledge-free adaptation, which uses no collected external knowledge.
Conclusion: The reported experiments show that ELEVATER’s collected external knowledge is useful for improving adaptation of language-augmented visual models.The authors identify more sophisticated knowledge-adaptation strategies as future work.