PaperlayerView exact revision on arXiv

Source-linked AI summary

On Large Multimodal Models as Open-World Image Classifiers

Alessandro Conti, Massimiliano Mancini, Enrico Fini, Yiming Wang, Paolo Rota, Elisa Ricci

arXiv:2503.21851v2cs.CV

TL;DR

The paper addresses the limited evidence on LMM image classification beyond predefined categories. It formalizes and evaluates open-world classification, finding varied metric strengths, substantial error types, and a remaining gap with closed-world models.

  • Problem

    Existing image classification studies typically assume fixed, known categories, limiting evidence about LMM performance when models can predict unconstrained semantic concepts.

  • Method

    The paper formalizes open-world classification, introduces four complementary alignment metrics, and evaluates 13 models across 10 benchmarks spanning multiple category granularities.

  • Results

    LMMs generally outperform contrastive models in open-world classification, leading in 11/16 metric/groups, but exhibit granularity and fine-grained errors and remain behind closed-world models.

  • Takeaways & Limitations

    Combining metrics reveals distinct error sources, while tailored prompting and reasoning can reduce overly generic and fine-grained classification mistakes.

  • Takeaways & Limitations

    Despite promising results, LMMs still show a large performance gap with closed-world models such as CLIP and SigLIP.

Abstract

from arXiv · show

Traditional image classification requires a predefined list of semantic categories. In contrast, Large Multimodal Models (LMMs) can sidestep this requirement by classifying images directly using natural language (e.g., answering the prompt "What is the main object in the image?"). Despite this remarkable capability, most existing studies on LMM classification performance are surprisingly limited in scope, often assuming a closed-world setting with a predefined set of categories. In this work, we address this gap by thoroughly evaluating LMM classification performance in a truly open-world setting. We first formalize the task and introduce an evaluation protocol, defining various metrics to assess the alignment between predicted and ground truth classes. We then evaluate 13 models across 10 benchmarks, encompassing prototypical, non-prototypical, fine-grained, and very fine-grained classes, demonstrating the challenges LMMs face in this task. Further analyses based on the proposed metrics reveal the types of errors LMMs make, highlighting challenges related to granularity and fine-grained capabilities, showing how tailored prompting and reasoning can alleviate them.

1. Introduction

The paper reframes image classification as an open-world task for LMMs, where models can name unconstrained semantic concepts rather than choose from fixed categories. It formalizes evaluation and uses large-scale benchmarking to expose both promising semantic alignment and persistent granularity and fine-grained errors.

  • 1. Introduction: Across the evaluation, LMMs often produce semantically related concepts and outperform contrastive open-world alternatives, yet remain far behind closed-world models.The paper uses multiple metrics to analyze errors involving generic predictions, similar concepts, and labeling ambiguities.
  • 1. Introduction: LMMs can answer open-ended image prompts and recognize concepts beyond a predefined category list, removing the closed-world constraint.Closed-world classification fixes a known label set, whereas open-world prediction allows unconstrained semantic concepts.
  • 1. Introduction: Existing LMM classification studies largely assume closed-world labels or use limited exact-label metrics that miss valid synonyms, granularity mismatches, and genuine confusions.Examples include sofa versus couch, can versus trash can, and bedroom versus bed.
  • 1. Introduction: The work formalizes open-world LMM classification and introduces four complementary metrics covering textual, judged, semantic, and concept-level alignment.The metrics are text inclusion, Llama inclusion, semantic similarity, and concept similarity.
  • 1. Introduction: The study evaluates 13 LMMs across 10 benchmarks spanning prototypical, non-prototypical, fine-grained, and very fine-grained categories.The evaluation covers datasets such as Caltech101, DTD, Flowers102, and Stanford Cars.
  • 1. Introduction: Metric disagreements identify error sources: tailored prompting helps overly generic predictions, while reasoning strategies address confusion among similar categories.The analysis also examines whether apparent mistakes reflect the single-label nature of benchmark datasets.

2. Related Work

Related work covers the evolution of large vision-language models from shared embedding spaces toward generative multimodal systems. It also situates this study among efforts to improve multimodal reasoning, fine-grained discrimination, and labeling.

  • 2. Related Work: Large vision-language models have progressed from shared visual-text embedding spaces to generative models producing text or multimodal outputs.Architectural differences include modality-alignment strategies such as MLP projectors and Q-formers, alongside autoregressive or alignment-based pretraining.
  • 2. Related Work: Prior research examines multimodal reasoning, fine-grained discrimination, and labeling, while this work focuses on the mistakes LMMs make during open-world image classification.The stated aim is to provide a foundation for improving LMM performance in this setting.

3. Benchmarking LMMs in OW Classification

The paper formalizes open-world image classification with LMMs and evaluates it using complementary metrics across diverse dataset groups. Results show strong performance on prototypical classes but substantial degradation on non-prototypical and fine-grained tasks, while joint metric analysis exposes distinct error types.

  • Metrics: The evaluation uses text inclusion, Llama inclusion, semantic similarity, and concept similarity to capture complementary forms of prediction–label alignment.TI checks string inclusion, LI uses Llama to judge alignment, SS measures embedding similarity, and CS compares sentence-part concepts.
  • Datasets and models: The benchmark spans coarse, non-prototypical, fine-grained, and very fine-grained datasets, with CLIP and SigLIP closed-world models plus two CLIP-based open-world retrieval baselines.The baselines retrieve captions or generate candidate classes from retrieved captions, while the LMMs predict directly in an unconstrained output space.
  • Benchmark results: LMMs perform best on prototypical classes, but performance drops substantially for non-prototypical, fine-grained, and very fine-grained classification.Non-prototypical LI reaches at most 46.8 with average CS of 49.3, while very fine-grained datasets often produce TI scores of 0.0.
  • Benchmark results: The study reports a remaining gap with closed-world CLIP and SigLIP models despite promising open-world LMM results.This comparison motivates further analysis of what the proposed metrics capture in open-world predictions.
  • Interpreting predictions: Joint analysis of inclusion and similarity distinguishes correct generic predictions, wrong specific predictions, and wrong generic mistakes.Examples include dog versus pug for correct but generic output and pug versus pomeranian for semantically similar but wrong output; LI and CS are thresholded to define the groups.

4. Analyzing LMMs Mistakes in OW

The analysis examines agreement, granularity control, reasoning, and whether models identify relevant image content in open-world classification. It finds strong generic recognition but persistent difficulty with fine-grained specificity and similar concepts.

  • Agreement across models: Only 5.6% of samples are correctly predicted by all models, while six fine-grained labels are never predicted correctly under LI.Agreement is higher on prototypical datasets, such as C101, than on very fine-grained datasets including CARS and FGVC.
  • Agreement across models: Models from the same family share many correct and specific predictions, including 66.7% for Qwen2VL 2B and 7B and 59.0% for InternVL2 4B and 8B.LLaVA NeXT with Mistral and Vicuna also share 63.9% of correct predictions.
  • Prompting for granularity: Generic prompting shifts predictions toward correct but generic responses, producing +9.1% good generic answers while reducing correct and specific ones.Specific prompting instead increases both wrong but specific and wrong and generic predictions, indicating weaker specificity.
  • Reasoning strategies: Test-time reasoning improves correct and specific responses, with Qwen2VL reaching gains of up to +13%.Zero-shot CoT and LLaVA-CoT consistently increase accuracy, whereas a multi-turn prompt also raises wrong and generic predictions by +1.7.
  • Reasoning strategies: Reasoning-specialized model replacements produce mixed results, increasing correct predictions by +6.6% but semantically close misclassifications by +3.7%.The comparison suggests test-time reasoning is more effective for these nuanced errors than replacing base models with reasoning variants.
  • Relevant but non-ground-truth content: Between 30% and 60% of wrong predictions match another image tag, showing that many errors identify relevant content despite differing from the ground-truth label.The analysis uses RAM++ tags and a concept-similarity threshold above 0.95 to identify such cases.

5. Conclusions

The conclusion presents a large-scale evaluation of LMMs for open-world classification and uses multiple alignment metrics to characterize their strengths and errors.

  • Conclusions: The study evaluates 13 models across 10 datasets with four metrics, identifying overly generic predictions, confusion between similar concepts, and attention to the wrong subject.The metrics capture different levels of alignment between predictions and ground truth.

A. Supplementary Material

The supplementary material extends the paper with dataset and model details, expanded results, model ranking, agreement analyses, and additional multi-label experiments.

  • Supplementary Material: The supplementary material reports dataset, model, metric, and per-dataset results, then adds Elo ranking, prediction-agreement analyses, and experiments on multiple candidate labels.It includes agreement analyses for correct but generic, wrong but specific, and wrong and generic predictions.

A.1. Additional details on the datasets and models

The evaluation uses established training and test splits and summarizes the datasets and LMMs used in the study.

  • Additional details on the datasets and models: Experiments use the same training and test splits as previous work, with datasets summarized in Table 3 and model details in Table 4.The tables provide the study’s dataset and model inventory.

A.2. Additional details on the metrics

The Llama inclusion metric uses Llama 3.2 to judge whether an LMM prediction is a good response relative to a target value.

  • Llama inclusion asks Llama 3.2 to determine whether an answer is a good reply to a question given its target value.The evaluator receives the question, answer, and target value.
  • The metric accepts descriptions of the target, explicit target or synonym matches, and generic targets related to the answer.The evaluator replies only with “1” for yes or “0” for no.

A.3. Extended results

The extended-results appendix reports per-dataset outcomes for the four evaluation metrics and summarizes the dataset collection used in the analyses.

  • Per-dataset results are reported separately for text inclusion, Llama inclusion, semantic similarity, and concept similarity.These results appear in Tables 5–8, respectively.
  • The appendix includes a summary table describing the datasets used in the analyses.

A.4. Additional analyses

Additional analyses compare model responses, model scale and provenance, prediction agreement, and factors associated with performance. They find strong Qwen2VL results, mixed effects of scaling, and family-level agreement patterns.

  • Model comparison: Qwen2VL models provide the best responses according to average Elo scores across the ten datasets.The Elo comparison evaluates model generations pairwise and reports average scores in Table 9.
  • Prediction agreement: Models from the same families tend to agree on the same samples across correct-but-generic, wrong-but-specific, and wrong-and-generic prediction groups.Figures 9–11 visualize pairwise agreement for these three error or outcome types.
  • Larger models: Scaling has mixed effects: InternVL2 26B and Qwen2VL 72B can improve performance, whereas InstructBLIP 13B and LLaVA-NeXT 34B can perform worse.
  • Commercial models: Open and commercial models show no large performance gap overall, although Claude consistently performs better than the compared open models.GPT models and Gemini are reported as comparable to InternVL2 26B and Qwen2-VL 72B.
  • Model comparison: Qwen2VL 7B outperforms Qwen2-VL 72B and commercial models on most metrics, while GPT-4o-mini exceeds GPT-4o and smaller models can match larger counterparts.The results also report comparable performance between some smaller and larger models.
  • Factors linked to performance: The analysis hypothesizes that vision-encoder pretraining, image resolution, training strategy, and training data matter more than language-encoder size alone.The hypotheses are illustrated through comparisons involving InternVL, Qwen2-VL, LLaVA, Idefics, InstructBLIP, and GPT models.

A.5. Extended results for the analyses

The appendix extends the prompting and reasoning analyses with larger and commercial models, reporting metric changes and shifts among correct, generic, and wrong prediction types.

  • Extended model results: Larger and commercial models are evaluated on grouped datasets using text inclusion, Llama inclusion, semantic similarity, and concept similarity.Tables 10 and 11 summarize these extended model comparisons.
  • Prompting and reasoning analyses: The appendix reports relative metric variation for multi-label, generic or specific, dataset-specific, and chain-of-thought prompts.Tables 12–14 and 16 cover these variations across selected datasets.
  • Prediction-type analyses: Figures 12 and 13 distinguish correct-specific, correct-generic, wrong-specific, and wrong-generic predictions for dataset-specific and multi-label prompting.Blue denotes correct outcomes and red denotes mistakes.
  • Prompting and reasoning analyses: Additional tables quantify gains in correct, generic, and wrong prediction types under generic or specific prompts and dataset-specific prompting.These analyses cover six datasets and include DTD, FGVCAircraft, Flowers102, Food101, OxfordPets, and StanfordCars.
  • Prompting and reasoning analyses: The appendix reports prediction-type gains for chain-of-thought reasoning, reasoning models, and multi-label prompts.Tables 17, 18, and 19 provide these results across the stated datasets.
Loading 2503.21851v2…