PaperlayerView exact revision on arXiv

Source-linked AI summary

African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification

Gregor Geigle, Radu Timofte, Goran Glavaš

arXiv:2406.14496v1cs.CVcs.CL

TL;DR

Fine-grained object classification is rarely tested in LVLMs, despite its importance for precise identification. The paper creates FOCI, a challenging multiple-choice benchmark using difficult CLIP-mined alternatives, and evaluates 12 public LVLMs. FOCI reveals a complementary skill to established benchmarks, with LVLM performance lagging behind the zero-shot accuracy of their CLIP encoders.

  • Problem

    LVLMs are rarely evaluated on fine-grained object classification, although precise identification matters for information-seeking tasks.

    Source: §1 · p. 1

  • Method

    FOCI converts existing classification datasets into multiple-choice tasks and uses CLIP to mine difficult incorrect labels.

    Source: §1 · p. 2, §3 · p. 4, §6 · p. 8

  • Results

    FOCI performance is largely uncorrelated with established benchmarks, and LVLMs lag drastically behind the zero-shot accuracy of their underlying CLIP encoders.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The authors suggest that LVLMs’ encoder–LLM alignment is insufficiently fine-grained and report that explicit object mentions in training captions can help classification.

    Source: §1 · p. 2

    The benchmark likely overestimates performance for diverse cultural objects and concepts because its datasets and image sources may bias it toward Anglospheric concepts.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent Large Vision-Language Models (LVLMs) demonstrate impressive abilities on numerous image understanding and reasoning tasks. The task of fine-grained object classification (e.g., distinction between \textit{animal species}), however, has been probed insufficiently, despite its downstream importance. We fill this evaluation gap by creating \texttt{FOCI} (\textbf{F}ine-grained \textbf{O}bject \textbf{C}lass\textbf{I}fication), a difficult multiple-choice benchmark for fine-grained object classification, from existing object classification datasets: (1) multiple-choice avoids ambiguous answers associated with casting classification as open-ended QA task; (2) we retain classification difficulty by mining negative labels with a CLIP model. \texttt{FOCI}\xspace complements five popular classification datasets with four domain-specific subsets from ImageNet-21k. We benchmark 12 public LVLMs on \texttt{FOCI} and show that it tests for a \textit{complementary skill} to established image understanding and reasoning benchmarks. Crucially, CLIP models exhibit dramatically better performance than LVLMs. Since the image encoders of LVLMs come from these CLIP models, this points to inadequate alignment for fine-grained object distinction between the encoder and the LLM and warrants (pre)training data with more fine-grained annotation. We release our code at \url{https://github.com/gregor-ge/FOCI-Benchmark}.

1 Introduction

Fine-grained object classification is rarely tested in LVLMs, despite its role in precise object recognition and information-seeking. The paper introduces FOCI, a difficult multiple-choice benchmark, and finds that its scores reveal capabilities not captured by established benchmarks and lag behind the underlying CLIP encoders.

1 Introduction

  • Fine-grained classification complements general image understanding, and precise identification can matter in information-seeking tasks; the dog example distinguishes misclassification from invented facts.LLaVA 1.5 gives information correct for a Samoyed, but the pictured dog is a Keeshond.Source: §1 · p. 1, Figure 1 · p. 1
  • FOCI uses multiple choice to specify admissible answers and taxonomy granularity, then mines difficult incorrect choices with a zero-shot CLIP model.This avoids ambiguity in open-ended answers while retaining challenge through semantically close alternatives.Source: §1 · p. 2
  • The benchmark combines five popular domain-specific classification datasets with four ImageNet-21k subsets covering animals, plants, food, and man-made objects.The five datasets cover flowers, cars, food, aircraft, and pets.Source: §1 · p. 2
  • Across 12 public LVLMs, FOCI performance is largely uncorrelated with established benchmarks, while CLIP encoders’ zero-shot accuracy upper-bounds LVLM performance, which lags drastically.The authors suggest this gap reflects insufficiently fine-grained semantic alignment between image encoders and LLMs.Source: §1 · p. 2
  • Larger LLMs and stronger image encoders improve FOCI results; explicit object mentions in captions and fine-grained classification objectives in training can also help.The authors suggest that the encoder–LLM alignment is not semantically fine-grained enough for object distinction.Source: §1 · p. 2

2 Related Work

Prior LVLM benchmarks mainly test general image understanding and reasoning, with fine-grained classification rarely evaluated. Existing classification approaches have limitations, while FOCI provides a standardized multiple-choice alternative and relates performance to LVLM training and alignment.

2 Related Work

  • LVLMs typically align a pretrained image encoder with an LLM in two training stages, and the first stage varies widely in training-data scale.The paper reports that better visio-linguistic alignment in the first stage substantially boosts fine-grained classification.Source: §2 · p. 2
  • FOCI’s multiple-choice setup selects the correct class and three CLIP-ranked incorrect labels, then asks the LVLM to choose the answer.The figure depicts CLIP similarity ranking, choice formulation, and LVLM prediction.Source: Figure 2 · p. 3
  • Common LVLM benchmarks emphasize image understanding and reasoning, while fine-grained object classification receives little evaluation.Prior approaches include few-shot comparisons with labeled examples, computationally expensive scoring of every class, and open-ended QA with ambiguous answers.Source: §2 · pp. 2–3

3 Multiple-Choice Image Classification

FOCI frames fine-grained classification as multiple choice using annotated datasets and CLIP-mined hard negatives. It combines established domain datasets with four domain-specific ImageNet-21k subsets.

3 Multiple-Choice Image Classification

  • FOCI frames fine-grained classification as a multiple-choice task with well-defined candidate answers.The task targets distinctions among objects within a specific domain, such as animal species or car models.Source: §3 · p. 3
  • Multiple choice avoids open-answer ambiguity about both the expected taxonomy granularity and the complete set of admissible labels.For example, several synonymous labels or labels at different taxonomy levels may correctly describe the same object.Source: §3 · p. 3
  • For each image, OpenCLIP ViT-L/14 selects the three most similar incorrect class labels as negatives, alongside the correct label.Similarity is ranked by cosine similarity between the image embedding and text embeddings of class labels; this encoder was not used by the evaluated LVLMs.Source: §3 · pp. 3–4
  • FOCI includes five established datasets: FGVC-Aircraft, Flowers102, Food101, Oxford-Pet, and Stanford-Cars.Together, they cover aircraft types, flower species, dishes, cat and dog breeds, and car models.Source: §3 · p. 4
  • Four additional ImageNet-21k subsets group leaf classes by WordNet domain: Animal (1322), Plant (957), Food (563), and Artifact (2631).The classes are leaf nodes, preventing labels at different taxonomy levels from yielding multiple correct answers.Source: §3 · p. 4
  • Appendix D additionally evaluates general ImageNet-1k classification under image-distribution shifts and GeoDE objects photographed across regions.The authors note that more object types and domains could be added, while aiming for a reasonably diverse benchmark from which to extrapolate general fine-grained recognition ability.Source: §3 · p. 4

4 Evaluating Public LVLMs

The section evaluates public LVLMs on FOCI and compares their results with established benchmarks and the corresponding CLIP encoders. Fine-grained classification results differ from other benchmark rankings, and LVLM accuracy remains below CLIP accuracy, with gaps varying by domain.

4 Evaluating Public LVLMs

  • 12 public LVLMs with LLMs of ≤7B parameters were evaluated using randomly ordered four-choice options, with accuracy as the metric.The section also examines how LVLM performance relates to that of the underlying CLIP image encoders.Source: §4 · pp. 4–5

4.1 Results

  • FOCI rankings are much less correlated with GQA, MMBench, and MMMU results, supporting fine-grained classification as a complementary skill for LVLM evaluation.Qwen-VL performs strongly on FOCI but worse on standard benchmarks, while Phi-3-Vision shows the reverse pattern.Source: §4.1 · p. 5
  • ∼1.5B images of pretraining distinguish the two best FOCI models from LLaVA models trained with <1M image-text pairs; InstructBLIP results also suggest the LLM matters.The InstructBLIP comparison is described as somewhat inconclusive: Flan-T5-XL performs well, while Vicuna performs substantially worse under otherwise identical training.Source: §4.1 · p. 5

4.2 LVLM vs. Its Corresponding CLIP

  • <10% on IN-Artifact to 40-50% on Oxford-Pets, LVLM-to-CLIP accuracy gaps vary by dataset, with CLIP accuracy an apparent upper bound.The comparison uses the same four labels for CLIP and LVLMs.Source: Figure 3 · p. 5, §4.2 · p. 6

4.1 Results

  • Higher image resolution does not seem relevant for fine-grained classification, where dataset images typically show large, centered objects.The observational analysis cannot isolate the effects of the LLM and image encoder.Source: §4.1 · p. 5

4.2 LVLM vs. Its Corresponding CLIP

  • 25% accuracy is approached by all three analyzed LVLMs on ImageNet-1k examples that their corresponding CLIP models misclassify.The authors report that CLIP errors propagate to LVLMs and identify image-encoder selection as a key design decision.Source: Figure 4 · p. 6, §4.2 · p. 6

5 Controlled Experiments

Controlled experiments vary LLM size, image encoder, and training data. Larger LLMs and stronger encoders improve FOCI results, while explicit object names in training captions and object-focused task-mix training also help.

LLM Size

  • FOCI performance consistently improves across datasets as MobileVLM v2 scales from 1.7B to 3B to 7B LLM backbones.The models share otherwise identical architecture and training, isolating the effect of LLM size.Source: Figure 5 · p. 7, §5 · p. 7

Image Encoder

  • SigLIP improves performance across FOCI, while increasing CLIP resolution from 224px to 336px yields only a marginal ∼1-point average gain.The resolution change varies by object type, with gains over 5 points on Flowers102 and Food102 but a 5-point drop on Oxford-Pets.Source: Figure 6 · p. 7, §5 · p. 7

Changes to Task Mix Phase

  • Adding ImageNet as an open-ended QA task to the second task-mix training phase improved performance by 6% on the 500 ImageNet-1k classes seen in training (Train Half) and by 2 points on held-out Test Half images and FOCI.Source: §5 · p. 8

Changes to Pretraining

  • Over 2 accuracy points: skipping alignment-module pretraining reduced average FOCI performance by this amount.The ablation compares skipping pretraining with training the alignment module on image-text pairs.Source: Table 3 · p. 7, §5 · p. 8
  • 5% for seen ImageNet classes with Template captions versus ∼1.5-point with Synthetic captions; these caption changes had very limited effect on FOCI performance.Template captions explicitly name the image object, whereas Synthetic captions may use a broader label.Source: Table 3 · p. 7, §5 · p. 8

6 Conclusion

The paper introduces FOCI to evaluate fine-grained object classification with a multiple-choice benchmark built to avoid ambiguous answers while preserving difficulty. Results from 12 public LVLMs show that FOCI measures a complementary skill, with image-encoder quality and explicit object mentions in training data associated with performance.

Conclusion

  • FOCI contains 9 fine-grained multiple-choice datasets and mines difficult, semantically close incorrect choices with CLIP to retain classification difficulty.The formulation avoids the ambiguous admissible answers of open-ended object-classification QA.Source: §6 · p. 8
  • FOCI performance is largely uncorrelated with established image-understanding and reasoning benchmarks, identifying fine-grained classification as a complementary skill.The paper’s ablations identify image-encoder quality and explicit object mentions in training data as factors associated with FOCI performance.Source: §6 · p. 8

Limitations

The benchmark has limits in cultural coverage, task formulation, and model scale. Evaluation uses dataset test splits or ImageNet validation data, with selected procedural and sampling exceptions.

Limitations

  • The datasets and English-internet image sources likely bias FOCI toward Anglospheric concepts, so performance may be overestimated for diverse cultural objects and concepts.ImageNet is specifically noted as biased in its image sources and classes.Source: Limitations · p. 9
  • Multiple-choice results may not directly match open-ended use: confounder options can make some objects harder, while including the correct name can make others easier.The authors expect results across the two formulations to correlate.Source: Limitations · p. 9
  • Evaluation covers only public LVLMs with LLMs of 7B parameters or less; larger and proprietary models were excluded due to inference constraints or API costs.The stated reasons include compute or VRAM limits for larger models and the expense of over 100,000 API calls for proprietary models.Source: Limitations · p. 9

A Evaluation Details

  • Models use greedy decoding and are counted correct when their answer begins with the letter for the correct option.The evaluation follows each model’s official code for inference.Source: A Evaluation Details · p. 13
  • Prompts use a default question or a domain-specific variant for datasets such as ImageNet animals, plants, aircraft, and flowers.The prompt asks which choice is shown in the image.Source: A Evaluation Details · p. 13
  • Evaluation uses full test splits, or ImageNet validation when no public test split exists; ImageNet-21k subsets use 10 images per class to limit computation.The processed ImageNet-21k images are resized to 224×224px.Source: A Evaluation Details · p. 13

B Training Details

Training follows the LLaVA 1.5 protocol, using StableLM 2 Zephyr 1.6B and OpenAI CLIP ViT-L/14-224 by default. The models are trained with AdamW, cosine learning-rate decay, and LoRA for task-mix training.

B Training Details

  • The implementation follows the LLaVA 1.5 architecture and training protocol, with StableLM 2 Zephyr 1.6B as the LLM and OpenAI CLIP ViT-L/14-224 as the default image encoder.Training one model takes less than two days on a single NVIDIA RTX 3090.Source: B Training Details · p. 13
  • AdamW with cosine learning-rate decay is used, with learning rate 1e-3 and batch size 256 for pretraining, and 2e-4 and 128 for task-mix training.Weight decay is 0 in both phases.Source: B Training Details · p. 13
  • Task-mix training applies LoRA to all LLM weights with r = 64 and α = 128 instead of fine-tuning the full LLM.The pretraining and task-mix settings use separate learning rates and batch sizes.Source: B Training Details · p. 13

C LVLM Performance on Popular Benchmarks

The comparison with GQA, MMBench, and MMMU indicates that object-classification performance is an independent skill that does not directly correlate with these benchmarks.

C LVLM Performance on Popular Benchmarks

  • Object-classification performance is an independent skill that does not directly correlate with GQA, MMBench, or MMMU.The authors collate public results on these benchmarks for the models in Table 1.Source: C LVLM Performance on Popular Benchmarks · p. 13

D Additional Evaluation on More Datasets

The section evaluates LVLMs on general-object classification under image-distribution and geographic shifts, finding regional accuracy differences in GeoDE.

D Additional Evaluation on More Datasets

  • The evaluation considers general object classification under image-distribution shifts using ImageNet variants and GeoDE.The ImageNet variants cover difficult images, object renditions, and sketches.Source: D Additional Evaluation on More Datasets · pp. 13–14
  • LVLM rankings on ImageNet variants resemble FOCI rankings, and their accuracy changes qualitatively resemble those of their underlying CLIP models.The authors suggest that image encoders represent alternate object forms, such as sketches, similarly enough for LVLM recognition without extra training on different image types.Source: D Additional Evaluation on More Datasets · p. 14
  • African GeoDE images have 2–4 points lower accuracy than overall, while non-African regions are 0–3 points below Europe.European images consistently have the highest accuracy, although GeoDE is generally easy with high accuracy throughout; the paper links these regional disadvantages to geographic biases in image-encoder and LVLM training data.Source: D Additional Evaluation on More Datasets · p. 14

E More CLIP Results

The section reports conditional-accuracy results across datasets and provides additional raw results for MobileVLMv2 and trained models.

F Full Experiment Results

E More CLIP Results

  • Conditional accuracy is reported across datasets according to whether the CLIP image encoder correctly classifies each example in zero-shot.Figure 7 presents the corresponding results for LVLMs.Source: E More CLIP Results · p. 14, Figure 7 · p. 16

Related papers

Loading 2406.14496v1…