What Remains OOD? OODConceptBench for Open-World Concept Recognition in Vision-Language Models

1 INTRODUCTION

• Existing benchmarks focus on particular capabilities or failure modes, while fixed-label OOD evaluation leaves open which open-world concepts VLMs can recognize.Vision–language pretraining allows recognition beyond a single downstream task’s labels.Source: §1 · p. 1
• OODConceptBench evaluates open-world concept recognition with a hierarchical benchmark and fixed image–concept pairs.Figure 1 contrasts predefined-class OOD with model-specific concept recognition.Source: §1 · p. 1, Figure 1 · p. 2
• The benchmark covers 560 concept phrases across seven categories, and its harder L2 subset selects pairs with lower image–text similarity to their assigned concepts.L1 contains 30,752 images and L2 contains 14,285 images.Source: §1 · p. 2
• The study evaluates 37 VLMs across release years, model families, variants, and benchmark categories, finding widespread open-world recognition difficulties.The authors identify continued research as needed.Source: §1 · p. 2

2 RELATED WORK

• Conventional OOD evaluation fixes ID and OOD data; related approaches unify settings or jointly evaluate closed-set classification, open-set discrimination, domain detection, and classification.OpenOOD standardizes comparisons across OOD settings and detection methods.Source: §2 · p. 3
• MCM, NegLabel, and InterNeg improve OOD detection under specified settings but do not directly compare how pretrained models recognize the same concepts.The passage attributes these methods to Ming et al. (2022), Jiang et al. (2024), and Xu et al. (2026).Source: §2 · p. 3
• OODBench studies covariate shift in a fixed label space, while SugarCrepe and NaturalBench focus on compositional understanding and difficult natural examples.These benchmarks characterize shifts, specific capabilities, or difficult cases rather than cross-model recognition of the same open-world concepts.Source: §2 · p. 3

3 PROBLEM DEFINITION

• Because zero-shot prompting can recognize concepts beyond a predefined taxonomy, falling outside the fixed ID label set does not alone mean a VLM cannot recognize a concept.The evaluation therefore directly tests fixed image–concept pairs.Source: §3 · p. 3
• For an image–concept pair, the model-specific recognition boundary compares image–text similarity with that model’s recognition threshold.The definition names the similarity as s_m(x_i, a_i) and the threshold as τ_m.Source: §3 · p. 3
• A pair is recognized when its similarity exceeds the threshold; the evaluation uses image–concept cosine similarity and a model-specific Q95,m threshold.The same pair may be recognized by one model but not another.Source: §3 · p. 3

4 BENCHMARK CONSTRUCTION

• Benchmark construction starts with a concept vocabulary and candidate images for each concept, supporting evaluation on fixed image–concept pairs.The paper describes the construction as following its OOD definition.Source: §4.1 · p. 4, §4 · p. 4

4.1 TARGET VOCABULARY, IMAGE COLLECTION, AND INITIAL CHECKS

• The vocabulary contains 560 phrases balanced across seven diagnostic categories; images come from web search and generation, then undergo usability and format checks.Time-lagged and Typographic conflict are defined as concepts affected by temporal change and cases where textual and visual cues conflict, respectively.Source: §4.1 · p. 4

4.2 FILTERING RULES

• Three filtering rules remove candidates close to common-category and rare reference banks, then select lower-alignment pairs from L1 for L2.The rules use CLIP ViT-B/32; larger reference-bank scores indicate greater distance, while lower target-text similarity indicates weaker alignment.Source: §4.2 · p. 4
• The filtering thresholds are calibrated, and Table 1 summarizes the criteria and resulting benchmark sizes; a WordNet expansion check supports broad reference-vocabulary coverage.Rule 1 uses S1K, rule 2 uses S21K on rule 1 survivors, and rule 3 uses STxt on rule 2 survivors.Source: §4.2 · pp. 4–5, Table 1 · p. 5

Benchmark overview

• L1 and L2 differ in category composition, image-source composition, and concept coverage.Figure 3 summarizes category counts, source composition, and concept coverage across L0, L1, and L2.Source: §5 · p. 5, Figure 3 · p. 5

Category composition

• All seven categories appear in both layers, but their sizes differ: L1 counts range from 3,032 to 5,838 images, versus 1,023 to 3,972 in L2.Fine-grained / texture and Typographic conflict mark the L1 range; Cultural blindspots and Time-lagged mark the L2 range.Source: §5 · p. 5

Image sources and concept coverage

• L2 has a higher web-search image share than L1, while covering fewer target concepts: 62.32% versus 47.53% web-search images and 519 versus 553 concepts.L0 covers all 560 concept phrases.Source: §5 · p. 5

6 MODEL EVALUATION; 6.1 EVALUATION PROTOCOL

• The 37 models are evaluated on the same fixed L1 and L2 sets, with a shared calibration pool and prompt bank but a separate threshold for each model.The reference pool contains 2,000 images from four datasets, and the prompt bank uses the same “a photo of <concept>” template.Source: §6 · p. 5, §6.1 · pp. 5–6

6.2 L2 RECOGNITION REMAINS LOWER THAN L1 ACROSS THE EVALUATED MODELS

• Under the fixed construction model, prediction entropy rises from 3.724 on L1 to 3.757 on L2, while target-semantic loss rises from 0.697 to 0.736.These diagnostics describe the fixed construction model; cross-model evidence for the L1/L2 ordering comes from model- and category-level recognition.Source: §6.2 · pp. 6–7
• Recognition on L2 is lower than on L1 for all 37 models, and 258 of 259 model–category comparisons show the same direction.Figure 4 compares each model's L1 and L2 recognition, grouped by family.Source: §6.2 · pp. 6–7, Figure 4 · p. 7

6.3 LATER-RELEASED MODELS DO NOT NECESSARILY SHOW STRONGER OPEN-WORLD TARGET-CONCEPT RECOGNITION

• Recognition does not increase steadily with release year: both layers decline from 2022 to 2023, and L1 also declines from 2024 to 2025.Figure 5(a) compares mean L1 and L2 recognition across release-year groups from 2021 to 2025.Source: §6.3 · p. 7, Figure 5 · p. 8

6.4 AN INCREASE IN OVERALL RECOGNITION DOES NOT IMPLY CONSISTENT IMPROVEMENT ACROSS ALL BENCHMARK CATEGORIES

• From 2024 to 2025, overall L2 recognition rises from 51.4% to 52.4%, while recognition falls in five categories; category-level changes also differ between L1 and L2 from 2021 to 2025.Across the seven categories, L2 recognition ranges from 24.6% for Fine-grained / texture to 68.9% for Synthetic / fictional.Source: §6.4 · pp. 8–9

6.5 SUPPORTING ANALYSES

• Supporting analyses indicate that the L1/L2 recognition gap is not simply explained by fixed-ID OOD behavior, semantic distance from ImageNet-1K, or a lack of learnable structure.The analyses compare fixed-ID scores, nearest-ImageNet-1K semantic similarity, and limited-supervision classification on frozen CLIP features.Source: §6.5 · p. 9

7 CONCLUSION

• Across all 37 models, target-concept recognition is lower on L2 than on L1.The reported analyses also find that recognition does not improve monotonically across release-year groups, and overall gains can coincide with declines in individual categories.Source: §7 · p. 9
• OODConceptBench evaluates whether a model recognizes an assigned concept on a fixed image–concept pair using a model-specific recognition boundary.This differs from defining OOD solely by membership in a predefined label set.Source: §7 · p. 9
• Supporting analyses suggest the observed L1 and L2 difficulty is not simply explained by fixed-ID OOD behavior, ImageNet-1K semantic distance, or benchmark-sample learnability.Source: §7 · p. 9

LLM USAGE

• Generative AI assisted with language editing, manuscript drafting and revision, document organization, literature retrieval, and reference checking.Source: LLM USAGE · p. 10
• The tools also provided feedback on experimental design and result interpretation, and helped generate and review data-processing, evaluation, and plotting code.Source: LLM USAGE · p. 10
• Image-generation tools created part of the benchmark’s generated-image component, and the authors independently verified research artifacts and take responsibility for the final content.The authors state that generative AI tools were not used to prove mathematical claims.Source: LLM USAGE · p. 10

ETHICS STATEMENT

• Public image-level release is conditioned on near-duplicate, sensitive-content, licensing, source-policy, and privacy review.Source: ETHICS STATEMENT · p. 10
• Generated images will be clearly marked, web-image source information retained, and local paths and other environment-specific information removed from released research files.Source: ETHICS STATEMENT · p. 10

Appendix D — Supporting Analyses

• Appendix E summarizes the benchmark’s intended use, release conditions, and study limitations.Source: Appendix D · p. 12

A.2 FILTERING RULE 1 AND FILTERING RULE 2 CALIBRATION RESULTS

• Filtering rules 1 and 2 use the 95th percentile of their calibration distributions; their thresholds are T1 = 0.7175 and T2 = 0.2976.KDE curves visualize distribution shape but do not determine either threshold.Source: §A.2 · p. 13, Figure A1 · p. 13

A.3 FILTERING RULE 3 CALIBRATION STABILITY

• Filtering rule 3’s full-vocabulary mean is 0.303085, reported as construction threshold T3 = 0.3031; steps 20 and 50 both use 200 prompts.Calibration used five rounds and sampling steps {1,2,3,5,8,10,20,50}.Source: §A.3 · pp. 13–14, Table A2 · p. 14, Figure A2 · p. 14

B.1 FINITE-SAMPLE BACKGROUND EXCEEDANCE CONTROL

• Under exchangeable background images, the finite-sample control applies to the calibration distribution and does not require independence among scores for different target prompts.With no ties almost surely, the first inequality is an equality.Source: §B.1 · p. 16

B.2 INVARIANCE TO STRICTLY INCREASING SCORE TRANSFORMATIONS

• Strictly increasing score transformations preserve every image–concept recognition decision when calibration and evaluation both use the transformed scores.The result also preserves reported recognition rates.Source: §B.2 · p. 16

C.4 CROSS-MODEL CATEGORY SUMMARY

• 258 of 259 paired model–category slices have lower recognition on L2 than L1; the exception is a near tie for DataComp-CLIP B/32 S on time-lagged concepts.The full category results are organized by model family for L1 and L2.Source: Table C2 · p. 18, Table C3 · p. 19, §C.4 · p. 20

C.5 RELEASE-YEAR AND VARIANT ANALYSES

• The 2021 and 2025 release-year groups are not matched causal samples; Table C7 reports model-weighted target-concept recognition by category at both benchmark levels, with differences calculated as 2025 minus 2021.For Table C5, year groups differ in family, data, objective, resolution, and backbone composition.Source: Table C5 · p. 20, Table C7 · p. 21

D.1 FIXED-ID REFERENCE RESULTS; D.6 SEMANTIC RELATION TO IMAGENET-1K

• Several diagnostic conclusions are reference- or construction-dependent: fixed-ID separation depends on the selected reference, and the lower L2 target-similarity mean is not independent validation.Nearest-ImageNet similarity uses a fixed CLIP ViT-B/32 text space; its reference is not a universal semantic measure.Source: §D.1 · p. 24, Table D1 · p. 24, §D.6 · p. 29

D.4 LOCAL THRESHOLD SENSITIVITY

• Local threshold changes yield Jaccard ranges of 0.348–0.938 for T1, 0.925–0.987 for T2, and 0.728–0.944 for T3; joint changes yield 0.312–0.876.This local sensitivity analysis does not establish global threshold optimality.Source: §D.4 · p. 27, Table D6 · p. 27, Figure D4 · p. 28

• Comparisons across the 37 evaluated models are descriptive because they differ along multiple dimensions, so observed differences cannot be attributed to a single factor.Source: §E.2 · p. 30
• The benchmark covers 560 target concept phrases across seven categories, so results should not be generalized to every possible concept.Source: §E.2 · p. 30
• L1/L2 membership depends on the fixed construction model, reference vocabularies, and thresholds, while model-specific Q95,m decisions depend on fixed prompts and reference calibration.Alternative prompt or calibration choices may change individual measurements; the fixed protocol provides a reproducible comparison basis.Source: §E.2 · p. 30